Feature #61542

Add two-digit ISO 639-1 keys to sys_language

Added by Benni Mack over 4 years ago. Updated over 1 year ago.

Status:
Closed
Priority:
Should have
Assignee:
Category:
Localization
Target version:
Start date:
2014-09-11
Due date:
% Done:

100%

PHP Version:
Tags:
Complexity:
Sprint Focus:

Description

The language handling of records in TYPO3
is solely based on UIDs of the sys_language DB table,
but no reference to the real language used.

The ISO 639-1 defines the language identifiers
(ISO language code) completely (182 entries).

If the real language key was needed before in
TYPO3, one could install static_info_tables
which takes a field ("static_lang_isocode") in the
various places of the TYPO3 Core and fetches
the isocode via separate SQL-Queries.

The change introduces the ISO language
two-letter-keys natively in the core in order to
1. use less SQL queries in FE and BE if
static_info_tables was installed
2. remove hard coded dependencies of
EXT:static_info_tables
3. and always ensure that the core
includes the ISO code

Additionally one can now use the
$TSFE->sys_language_isocode all the time for
working with the ISO code instead of the
UID parameter, meaning that isocode can now be
set with a TypoScript parameter as well.

A Migration Wizard moves values put in the old
DB field (which is a UID reference to
static_languages) to the new DB field "iso2code".

The old field is deprecated to use without the
new field filled "iso2code".

In TYPO3 CMS 8 the static_lang_isocode field
will be removed within the CMS core.


Related issues

Related to TYPO3 Core - Bug #22119: Include ISO_3166-1 alpha2 in sys_languages Closed 2010-02-14
Related to TYPO3 Core - Task #68616: Upgrade Wizard without installed ext:static_info_tables failes Closed 2015-07-29
Related to TYPO3 Core - Task #69929: Unsorted list of ISO codes Closed 2015-09-18

Associated revisions

Revision f1e399f3 (diff)
Added by Benni Mack about 4 years ago

[DB][FEATURE] Add ISO 639-1 keys to sys_language

The language handling of records in TYPO3
is solely based on UIDs of the sys_language DB table,
but no reference to the real language used.

The ISO 639-1 defines the language identifiers
(ISO language code) completely (182 entries).

If the real language key was needed before in
TYPO3, one could install static_info_tables
which takes a field ("static_lang_isocode") in the
various places of the TYPO3 Core and fetches
the isocode via separate SQL-Queries.

The change introduces the ISO language
two-letter-keys natively in the core in order to
1. use less SQL queries in FE and BE if
static_info_tables was installed
2. remove hard coded dependencies of
EXT:static_info_tables
3. and always ensure that the core
includes the ISO code

Additionally one can now use the
$TSFE->sys_language_isocode all the time for
working with the ISO code instead of the
UID parameter, meaning that isocode can now be
set with a TypoScript parameter as well.

A Migration Wizard moves values put in the old
DB field (which is a UID reference to
static_languages) to the new DB field "language_isocode".

The old field is deprecated to use without the
new properly filled field "language_isocode".

In TYPO3 CMS 8 the static_lang_isocode field
will be removed within the CMS core (but could
still be included in static_info_tables).

Releases: master
Resolves: #61542
Change-Id: Ia017af52af354ac204ffac11635d99df120b029a
Reviewed-on: http://review.typo3.org/32884
Reviewed-by: Markus Klein <>
Tested-by: Markus Klein <>
Reviewed-by: Georg Ringer <>
Tested-by: Georg Ringer <>

Revision c3e9df61 (diff)
Added by Markus Klein about 4 years ago

[BUGFIX] Add missing identifier to newly introduced xliff file

Resolves: #61542
Releases: master
Change-Id: I18f3d43b3f7c868f68d935676d3896632dafc1e6
Reviewed-on: http://review.typo3.org/35022
Reviewed-by: Markus Klein <>
Tested-by: Markus Klein <>

Revision 2ec2acb4 (diff)
Added by Morton Jonuschat over 3 years ago

[BUGFIX] Install Tool: Make ISO language SQL queries DBAL compatible

Use database independent quoting for field values and compare the
numeric static_lang_isocode field with an integer instead of an empty
string.

Resolves: #67707
Related: #61542
Releases: master
Change-Id: I7c7e1352e92598a57524a05c9e048391df63df13
Reviewed-on: http://review.typo3.org/40575
Reviewed-by: Markus Klein <>
Reviewed-by: Benjamin Mack <>
Tested-by: Benjamin Mack <>
Reviewed-by: Andreas Fernandez <>
Tested-by: Andreas Fernandez <>

Revision a87e2b29 (diff)
Added by Xavier Perseguers over 3 years ago

[TASK] Sort the list of ISO codes

Resolves: #69929
Related: #61542
Releases: master
Change-Id: I002628c9741860d47c6fc6921ffb08b8ed256a70
Reviewed-on: http://review.typo3.org/43412
Reviewed-by: Frank Nägler <>
Tested-by: Frank Nägler <>
Reviewed-by: Andreas Fernandez <>
Tested-by: Andreas Fernandez <>

History

#1 Updated by Stefan Neufeind over 4 years ago

Sounds great. Didn't we also even discuss going one step further to allow "arbitrary" languages to be defined, also with country-specific additions (like de_CH) or completely "invented" languages for other use-cases? Can this be integrated with this change maybe?

#2 Updated by Sigfried Arnold over 4 years ago

i suggested this some time ago, but i don't think this is a good idea

if you go for i19n, then ISO 639-1 is the right way to do - in most cases ...

but lots of "international" sites are in fact localized (l10n) - so ISO 639-1 would not be enough, this need to be combined with ISO 3166 Alpha 2 to be compilant to RFC 3066

For example "english" could be represented in multiple forms

en (english, plain)
en-US (english, targeted to audience in the USA)

this is important in countries with multiple languages for example switzerland - de-CH significantly differs from de-AT or de-DE

even more "problematic" is, that TYPO3 mixes things up and uses both:
db fields are sometimes named i18n_ - for example "pages.I18n_cfg" or "pages_language_overlay.I18n_diffsource" where other tables are namend l10n_ - for example "sys_category.l10n_parent"

if this is added, it should be done like static_info_tables.

the issue #1 Benjamin Mack states (takes more SQL queries) is not true, since this can be resolved with a simple join - it still would be 1 query (bit a bit more complex, but this should not be an impact to the overall performance)

i guess it is important to let people choose if the want to i18n or l10n their website

#3 Updated by Daniel Maier over 4 years ago

We should also consider that in ISO 639-1 there are no separate languages codes for the different versions of the Chinese language.

ISO 639-1 language code "zh" is both used for traditional Chinese (mainly used for example in Taiwan) and simplified Chinese (mainly used in mainland China). As far as I know they are more than just different dialects; the differences between them seem bigger than differences between e.g German dialects; they for example also differ in the used characters.

What do you think about that? How could we handle this?

#4 Updated by Benni Mack over 4 years ago

Sigfried Arnold wrote:

i suggested this some time ago, but i don't think this is a good idea

For example "english" could be represented in multiple forms

en (english, plain)
en-US (english, targeted to audience in the USA)

this is important in countries with multiple languages for example switzerland - de-CH significantly differs from de-AT or de-DE

I agree here. However, I would consider combining this info with a separate field. One is stating the language (ISO 639-1) and the other the more specific locale.

even more "problematic" is, that TYPO3 mixes things up and uses both:
db fields are sometimes named i18n_ - for example "pages.I18n_cfg" or "pages_language_overlay.I18n_diffsource" where other tables are namend l10n_ - for example "sys_category.l10n_parent"

Yes. Another thing to resolve.

if this is added, it should be done like static_info_tables.

What do you mean? Adding it in a separate table? I feel like having a string in the DB (finally) for the isocode is finally going in the right direction. I don't see a benefit having this info in the DB for now.

the issue #1 Benjamin Mack states (takes more SQL queries) is not true, since this can be resolved with a simple join - it still would be 1 query (bit a bit more complex, but this should not be an impact to the overall performance)

Yes. I agree, but this is not how it is currently in the core (as static_info_tables is not part of the core).

#5 Updated by Gerrit Code Review over 4 years ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32884

#6 Updated by Gerrit Code Review over 4 years ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32884

#7 Updated by Gerrit Code Review over 4 years ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32884

#8 Updated by Gerrit Code Review over 4 years ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32884

#9 Updated by Gerrit Code Review over 4 years ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32884

#10 Updated by Gerrit Code Review over 4 years ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32884

#11 Updated by Gerrit Code Review over 4 years ago

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32884

#12 Updated by Martin Kutschker over 4 years ago

As real world sites use IETF language tags this issue (patch) should have follow-ups to add a second field for ISO 3166-1 alpha-2 country codes. An even more perfect solution (a third patch) would add also a field for ISO 15924 script codes.

So IMHO this is definitely a step in the right direction, it must not be the last one.

http://en.wikipedia.org/wiki/IETF_language_tag
http://en.wikipedia.org/wiki/ISO_3166-1
http://en.wikipedia.org/wiki/ISO_15924

#13 Updated by Gerrit Code Review over 4 years ago

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32884

#14 Updated by Gerrit Code Review over 4 years ago

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32884

#15 Updated by Gerrit Code Review over 4 years ago

Patch set 10 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32884

#16 Updated by Gerrit Code Review over 4 years ago

Patch set 11 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32884

#17 Updated by Gerrit Code Review over 4 years ago

Patch set 12 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32884

#18 Updated by Gerrit Code Review about 4 years ago

Patch set 13 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32884

#19 Updated by Gerrit Code Review about 4 years ago

Patch set 14 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32884

#20 Updated by Gerrit Code Review about 4 years ago

Patch set 15 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32884

#21 Updated by Benni Mack about 4 years ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

#22 Updated by Gerrit Code Review about 4 years ago

  • Status changed from Resolved to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35022

#23 Updated by Markus Klein about 4 years ago

  • Status changed from Under Review to Resolved

#24 Updated by Riccardo De Contardi over 1 year ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF