TYPO3 Core

TCA says "Requires frontend support". To me it seems extbase is just lacking this frontend support.

Dmitry, not sure wether I understood your use case properly. This is my use case:

As Developer I want to be able to assign a relation on a record in the default translation that must not be changed in the translation of the record.
When I use the \TYPO3\CMS\Extbase\Persistence\QueryInterface to lookup records by a category
Then I expect to find records in the language of the Frontend

//$TCA
'faculty' => array(
            'exclude' => 0,
            'l10n_mode' => 'exclude',
            'l10n_display' => 'hideDiff,defaultAsReadonly',
            'label' => 'LLL:EXT:luhjobs/Resources/Private/Language/locallang_db.xlf:tx_luhjobs_domain_model_vacancy.faculty',
            'config' => array(
                'type' => 'select',
                'foreign_table' => 'tx_luhjobs_domain_model_faculty',
                'foreign_table_where' => 'AND tx_luhjobs_domain_model_faculty.sys_language_uid IN (-1,0)',
                'items' => array(
                    array('',''),
                ),
                'minitems' => 1,
                'maxitems' => 1,
            ),
        ),

// QueryInterface
$constraints[] = $query->equals('faculty', $uid);

// resulting SQL(relevant parts only) in case of Frontend with ?L=1
tx_luhjobs_domain_model_vacancy.type = '19' AND tx_luhjobs_domain_model_vacancy.intern = 0
)
AND
(
    tx_luhjobs_domain_model_vacancy.sys_language_uid IN (1,-1)
    OR
    (
        tx_luhjobs_domain_model_vacancy.sys_language_uid=0 AND tx_luhjobs_domain_model_vacancy.uid NOT IN
            (
                SELECT tx_luhjobs_domain_model_vacancy.l10n_parent FROM tx_luhjobs_domain_model_vacancy WHERE tx_luhjobs_domain_model_vacancy.l10n_parent>0 AND tx_luhjobs_domain_model_vacancy.sys_language_uid=1 AND tx_luhjobs_domain_model_vacancy.deleted=0
            )
    )
)

Peter, I think you understood me correctly if I understood you correctly :)

There can be fields that should be changed in translated records (title is a typical example) and there can be fields that are common for the record in the default language and translated record. Those common fields are marked with 'l10n_mode' => 'exclude'. However it is not ight that Extbase lacks FE support. There was support but it was broken due to misunderstanding of TYPO3 localisation. It is quite easy to track where it was made.

For our own purposes I made it working correctly with an Xclass:

namespace Snowflake\Xyz\Xclass;

class Typo3DbQueryParser extends \TYPO3\CMS\Extbase\Persistence\Generic\Storage\Typo3DbQueryParser {

    /**
     * Fixes incorrect localisation handling in Extbase.
     *
     * @param string $tableName
     * @param array $sql
     * @param \TYPO3\CMS\Extbase\Persistence\Generic\QuerySettingsInterface $querySettings
     */
    protected function addSysLanguageStatement($tableName, array &$sql, $querySettings) {
        if (is_array($GLOBALS['TCA'][$tableName]['ctrl'])) {
            if (!empty($GLOBALS['TCA'][$tableName]['ctrl']['languageField'])) {
                $sql['additionalWhereClause'][] = $tableName . '.' . $GLOBALS['TCA'][$tableName]['ctrl']['languageField'] . '=0';
            }
        }
    }

}

Peter, https://forge.typo3.org/issues/62921#note-2 is not a solution but a workaround. You are trying to make incorrect localisation code in Extbase work properly. However it is not $TCA that should be changed but Extbase.

Check FE localisation guide and see that Extbase does it incorrectly.

I am against removing symptoms, I am for fixing the issue.

https://forge.typo3.org/issues/29293 can only be found through Google cache because it was in the MVC project and the project is deleted, so all its issues.

I think extbase does not respect/support "l10_mode" in TCA, at least I can't find any reference to "l10_mode" in extbase code or it's history. TYPO3-Core/TCA provides l10n_mode which results in a hugh difference in data handling, which extbase should be aware of and take into account.

Are you sure the code in your xclass covers all use cases of translations in a proper way?

But, you are probably right, the workaround with l10_mode='' probably has a huge problem when the relation of the original record is updated. The translation will still point to the old relation and due to l10n_display="defaultAsReadonly" there is no way to fix it. At least that's what I expect to happen without testing it.

There is no need for Extbase to do anything about l10n_mode. This happens in PageRepository::getRecordOverlay(). Extbase calls that function already.

And, yes, my code is correct. Again, see the FE localisation guide document by Kasper. I was a co-author and the first reviewer.

Are we talking about the table "pages" only, or are we talking about the generic approach in extbase to handle translations? The pages table is very special in that regard as you probably know.

Yes, for sure extbase must know about l10n_mode. It just results in a completly different data-handling of the core. As the data in the database is managed very different based on l10n_mode anyone trying to make sql-queries for the data must absolutly take the l10n_mode into account.

l10n_mode="exclude" ist even rather simple, but how would you expect the logic for l10n_mode="mergeIfNotBlank" to be implemented without teaching extbase about l10n_mode?

Peter, it does not matter what table it is. Extbase does NOT need to know about l10n_mode. Everything is handled by the core in PageRepository::getRecordOverlay() and related functions.

Look at https://git.typo3.org/Packages/TYPO3.CMS.git/blob/HEAD:/typo3/sysext/frontend/Classes/Page/PageRepository.php#l408 and https://git.typo3.org/Packages/TYPO3.CMS.git/blob/HEAD:/typo3/sysext/frontend/Classes/Page/PageRepository.php#l1427.

Core already does all that.

Extbase calls getRecordOverlay() here: https://git.typo3.org/Packages/TYPO3.CMS.git/blob/HEAD:/typo3/sysext/extbase/Classes/Persistence/Generic/Storage/Typo3DbBackend.php#l766

There is no any need for Extbase to care about l10n_mode because it is done by the core api. Look at the code!

Any more questions?

Best thing really would be to have proper test-scenarios to cover all different cases. Then we'd know for sure when it works.

Thx for drawing my attention again to the fact that extbase does use PageRepository::getRecordOverlay() to apply overlays to a record. This definitly does work fine.

However I am still not 100% convinced that it's really sufficient for lookups on the whole property chain, as the problem is not in applying the overlay. The problem start's in the beginning when trying to find the proper record to apply the overlay on.

Can we check that please once more really good?

Let's say we have a simple n:1 (select) relation like:

jobs.type => jobtypes

Now, I(extbase) needs to lookup all jobs where the type=3 and that should be displayed when the requested Frontend language is 1(non-default!).

Just looking for recordes with sys_language_uid=0 (that's what imho your xclass boils down to) will miss all records that are not translations but "standalone foreign language" records (having l10n_parent=0 and sys_language_uid=1).

I can only ask you to read again the FE localisation guide. There is no such thing as "standalone foreign language records" in TYPO3. TYPO3 always requires a record in the main language for having localized records. All other interpretations are wrong by design. Just read the doc: http://docs.typo3.org/typo3cms/FrontendLocalizationGuide/CoreSupportForLocalization/TcaTcemainTceforms/Index.html

What you write about, can only happen if somebody uses localisation in TYPO3 incorrectly.

Now we get somewhere else. The localisation guide imho does not explicitly state that TYPO3-Core does not support a "standalone foreign language record". The TYPO3-Core has always allowed to use the system in such a way without any exception or error message. So I'd consider it to be a valid use case and have always thought so.

If we can agree on this being a non-valid use case we should try to prevent it and can remove support for it. Do you think that's doable?

The TYPO3-Core has always allowed to use the system in such a way without any exception or error message. So I'd consider it to be a valid use case and have always thought so.

Well, you thought so but TYPO3 core does not really allow this :) You can create such records but they will only be visible in he List module, which lists everything with the goal to help detect all kind of issues. But all other TYPO3 core functions require records in the main language to work with localisations. There is no single function that supports "standalone" localised records.

Many people are mistaken and think that it is possible to create a record in any language and just use it. It does not work like that in TYPO3. There is always the main record and the overlay. Main record can be hidden in FE if it is unused but it has to exist. It is a strange but no more strange than other things in TYPO3 (see versions and workspaces, for example of other strange decisions).

If an extension uses records in the translated language directly, it means only one thing: its author is unaware of getRecordOverlay() and how core localisation works.

Hm,

So what is the reason for admins having to explicitly enable strict forcing the overlay thing?

// tsref
// http://docs.typo3.org/typo3cms/TyposcriptReference/Setup/Config/Index.html#sys-language-overlay
// 0 being the default
config.sys_language_overlay = 1

// Page-TSconfig
// http://docs.typo3.org/typo3cms/TSconfigReference/PageTsconfig/Mod/Index.html#web-page-mod-web-layout
mod.web_layout {
  defLangBinding = 1
}

Actually these settings are non-default and must be enabled. So to me it seems like whoever designed it had the most flexibilty in mind allowing different approaches including "standalone foreign language records" as well as "enforce translation/hide elements without a parent in default language".

Hi Dmitry,

first of all it's a pleasure for me to share knowledge with you. I am still very convinced it's just working, as I have been using TYPO3 this way with customers for almost 15 years now. Actually I ran a bunch of tests in BE and FE while writing all these comments. From practical point:

- standalone translated records in tt_content work
- standalone translated records are supported in page module in BE

There are options to disable this functionality (see my last comment), but to be honest I've always kept the flexibility that was available and requested by our customers. BTW the overlay api was imho not even existant in the very beginning(?) and got implemented/adjusted(?) with the WS/version feature, which was added some time later (by Kasper).

It would be way easier if TYPO3 was not as flexibel as it is, but hey, that's TYPO3 :) We can't even agree on wether this is a feature (my POW) or a bug (your POW). And, TYPO3 CMS's still powerfull and works. That's awesome, isn't it?

Should we go for a poll or an RFC on AC-ML?

Peter, tt_content is somehow the worst test case. What was mentioned before, the records in another language without relation to their "parent" record (= in the default language) is different with tt_content. It might look like it works, just because you actually do not browse the CE directly but their parent, the page with an URL. The desired effect might be totally different with other tables. So for tt_content is actually does not matter much. Correct me if I am wrong. Yes agree, this strict, overlay etc is pain in the ass for a beginning developer with his/her first EXT.

No, it does not look like it works, it definitly works, for almost fifteen years and as of today(tested on 6.2-dev). You can create content in any translation as you want, and it works in FE as you would expect it ("Out of the Box"). And I have adapted this working model for extensions/other records as well. List module afaik still allows to disable language view, hiding any parent/child translation relations. You won't convience me here. If there is a majority that shares your point of view, I am fine to adapt myself to it.

No, Peter, it cannot work for content out of the box :) Here the code that creates the query to fetch content: https://git.typo3.org/Packages/TYPO3.CMS.git/blob/HEAD:/typo3/sysext/frontend/Classes/ContentObject/ContentObjectRenderer.php#l8027

        if ($conf['languageField']) {
            // The sys_language record UID of the content of the page
            $sys_language_content = (int)$GLOBALS['TSFE']->sys_language_content;

            if ($GLOBALS['TSFE']->sys_language_contentOL
                && isset($GLOBALS['TCA'][$table])
                && !empty($GLOBALS['TCA'][$table]['ctrl']['languageField'])
                && !empty($GLOBALS['TCA'][$table]['ctrl']['transOrigPointerField'])
            ) {
                // Sys language content is set to zero/-1 - and it is expected that whatever routine processes the output will
                // OVERLAY the records with localized versions!
                $languageQuery = $conf['languageField'] . ' IN (0,-1)';
                // Use this option to include records that don't have a default translation
                // (originalpointerfield is 0 and the language field contains the requested language)
                if (!empty($conf['includeRecordsWithoutDefaultTranslation'])) {
                    $languageQuery .= ' OR (' . $GLOBALS['TCA'][$table]['ctrl']['transOrigPointerField'] . ' = 0 AND ' .
                        $conf['languageField'] . ' = ' . $sys_language_content . ')';
                }
            } else {
                $languageQuery = $conf['languageField'] . ' = ' . $sys_language_content;
            }
            $query .= ' AND (' . $languageQuery . ')';
        }

• you set select.includeRecordsWithoutDefaultTranslation = 1
  or
• you unset $TCA['tt_content']['ctrl']['languageField'] or 'transOrigPointerField'
• you set select. languageField in your configuration

However the case above is completely different from this bug report because it is about a custom configuration of the content object.

Do you use a modified core like most companies do? Or do you use fluidcontent, which depends on Extbase and uses the same wrong code?

The problem with Extbase is that it fetches the translated record record directly. So the overlay code, which is called after, is wrong. Here is the commit that broke it: https://review.typo3.org/#/c/10188/

Yes agree, this strict, overlay etc is pain in the ass for a beginning developer with his/her first EXT.

I fully agree with this. I hate this scheme and I would prefer standalone records myself. But if they are implemented, support for l10n_mode is required!

Hi,

You are aware of the fact that $GLOBALS['TSFE']->sys_language_contentOL is unset by default?

// have a look at
TypoScriptFrontendController->settingLanguage()
$this->sys_language_contentOL = $this->config['config']['sys_language_overlay'];
// $this->sys_language_contentOL is intialized with 0 in Controller
// $this->config['config']['sys_language_overlay'] is NOT set by default
// so we end up with $this->sys_language_contentOL = NULL

So for the language part of the query we get:

$languageQuery = $conf['languageField'] . ' = ' . $sys_language_content;

Testing on the most plain TS implementation I could think of with a pristine 6.2-dev branch:

page.50 < styles.content.get

[globalVar = GP:L = 1]
config.sys_language_uid = 1
config.language = en
config.locale_all = en_GB.UTF-8
[global]

Now, of course the whole story does get different when I set

config.sys_language_overlay = 1
//

This whole thing probably needs some cleanup/rethinking but I am quite sure it does work like I describe.

Just one additional note to your proposed patch on extbase query-parser. Regardless of our current focus of the discussion of the language overlays, as soon as the relation can be edited on a translation of a record the patch will fail as well. When you search on a property.path of a domain model (a relation), you must take into account that the translated record could have a different relation as the original one. Unless we really simplify this as well and make relations editable only on original records (Which definitly won't be sufficient for workspace/version OL)

Or, to put it simple: The whole idea/concept of querying in the default language and applying overlays afterwards is terribly broken as soon as you run queries across relations. Give me all news records in english translation that have category 18 assigned. There is no way to get this working by querying only the default language and applying overlays afterwards. Unless you make the restriction that the relation can't be set on the translated record.

I agree that it is broken. The idea is bad from the beginning. However this is official way. And when you specify l10n_mode=exclude to manage those categories only on the main language record, you expect it to work. It doesn't.

We are not talking here about concepts. This bug is only about making Extbase work properly with l10n_mode=exclude. If you want to change localisation concept, open another bug report, please.

It's broken, we agree. We don't agree on the official way, which has an high impact on the approach to fix the problem. I'd like to keep this Bug-Item for reference to get this thing fixed as it has a great valuable discussion and a lot of background knowledge. My fix/suggestion will however be very different from the solution you suggest. I can setup another issue if you want to keep this one for yourself, just let me know.

I prefer that you make another issue. This issue is very specific. If you want to make more generic change, it is not for this issue.

Not fixed I think. Otherwise it would be green.

You can still use the patch locally though.

Similar issue: #57272

My original issue: https://forge.typo3.org/issues/76519

Always a good example is the news extension:

'categories' => [
    'exclude' => 1,
    'l10n_mode' => 'mergeIfNotBlank',
    'label' => $ll . 'tx_news_domain_model_news.categories',
    'config' => [
        'type' => 'select',
        'renderType' => 'selectTree',
        'treeConfig' => [
            'dataProvider' => \GeorgRinger\News\TreeProvider\DatabaseTreeDataProvider::class,
            'parentField' => 'parent',
            'appearance' => [
                'showHeader' => true,
                'allowRecursiveMode' => true,
                'expandAll' => true,
                'maxLevels' => 99,
            ],
        ],
        'MM' => 'sys_category_record_mm',
        'MM_match_fields' => [
            'fieldname' => 'categories',
            'tablenames' => 'tx_news_domain_model_news',
        ],
        'MM_opposite_field' => 'items',
        'foreign_table' => 'sys_category',
        'foreign_table_where' => ' AND (sys_category.sys_language_uid = 0 OR sys_category.l10n_parent = 0) ORDER BY sys_category.sorting',
        'size' => 10,
        'autoSizeMax' => 20,
        'minitems' => 0,
        'maxitems' => 99,
    ]
],

Let's say I have a news entry in the default langauge with some categories selected.
Now I translate that news to another language.

The expected behaviour for me would be that the categories are inherited from the original article, but they're empty in the frontend.
Naturally, when I add categories to the translated article, those are shown in the frontend.

Are you sure this only affects relations? I've a table with several l10n_mode=exclude fields (integers). With 7.6, a query which compares against this fields ($query->equals()) results in no records are beeing found at all when sys_language_uid > 0. In 4.5 it used to work...

@Kornblum: Yes broken. You can try my core patch https://review.typo3.org/49332 if it helps you.