TYPO3 Core

I can reproduce this in TYPO3 6.2 and with an own extension. Images are just not handled correctly, and in the translation of a record it should be possible to remove images or add new ones. But ONLY the images of the default record are shown in the frontend.

When using...

'l10n_mode' => 'noCopy'

...in TCA, the fields are actually not copied and not shown in the frontend when displaying the translated record, but if I manually add another image to the translated record, the default language images are shown again. So I guess this is, apart from other translation handling issues in extbase/TYPO3 a major bug.

I also tried:

'l10n_mode' => 'mergeIfNotBlank'

...but it doesn't work and only the default record images are shown.

A multilanguage CMS should really work with multiple languages IMHO.

Related to #59192

FAL image localization problem faced in the case of flexforms also. I've created a flexform element based on this http://wiki.typo3.org/File_Abstraction_Layer#FlexForm, but it has a problem when translating default content element records. When I translate default content element, the images are not copied to translated element. If we put "media" as the fieldname in flexform, it copied the images, but it shows some unwanted images along with that.

Workaround for the original problem:

Within \TYPO3\CMS\Extbase\Persistence\Generic\Mapper\DataMapper::getConstraint() there is an equals-constraint with "$parentObject" - this is the object that has the FAL-relation as one of its properties. When the equals-constraint is evaluated, this constraint will be transformed into a SQL-where with the uid of "$parentObject" and that´s the reason why only the FAL-relations of the not-localized object will be used, the uid of the localized object will be ignored, but the FAL-relations of the localized objects are stored with the uid of the localized objects.

There are different ways to hotfix:
You can just extend DataMapper, inject your class with the "config.tx_extbase.objects"-TypoScript and override getConstraint() to use "$parentObject->_getProperty('_localizedUid')" in the equals-constraint.
For a less global change you could check the table and field names within getConstraint() and restrict this change to the fields where you need it.
A more object oriented approach would be to also extend AbstractDomainObject, write some kind of "constraint uid getter", integrate this method within DataMapper::getConstraint() and return the "_localizedUid" property in the model classes where required - you could also pass the fieldname as an argument.

as first solution, following Robert suggestions, i solved the problem modifyng \TYPO3\CMS\Extbase\Persistence\Generic\Mapper\DataMapper::getConstraint()

line

$constraint = $query->equals($columnMap->getParentKeyFieldName(), $parentObject);

becomes

if ($columnMap->getColumnName() == 'fal_related_files') {
    $constraint = $query->equals($columnMap->getParentKeyFieldName(), $parentObject->_getProperty('_localizedUid'));
} else {
    $constraint = $query->equals($columnMap->getParentKeyFieldName(), $parentObject);
}

update: we can apply this temporary patch as a XCLASS (using it on 6.2.4)

in ext_localconf.php:

// XCLASS
// fix https://forge.typo3.org/issues/57272
$GLOBALS['TYPO3_CONF_VARS']['SYS']['Objects']['TYPO3\\CMS\\Extbase\\Persistence\\Generic\\Mapper\\DataMapper'] = array(
    'className' => 'Tx_Myext_XClass_DataMapper'
);

and then in /Classes/XClasses/DataMapper.php

<?php

/**
 * A mapper to map database tables configured in $TCA on domain objects.
 */
class Tx_Myext_XClass_DataMapper extends \TYPO3\CMS\Extbase\Persistence\Generic\Mapper\DataMapper
{

    /**
     * Builds and returns the constraint for multi value properties.
     *
     * @param \TYPO3\CMS\Extbase\Persistence\QueryInterface $query
     * @param \TYPO3\CMS\Extbase\DomainObject\DomainObjectInterface $parentObject
     * @param string $propertyName
     * @param string $fieldValue
     * @param array $relationTableMatchFields
     * @return \TYPO3\CMS\Extbase\Persistence\Generic\Qom\ConstraintInterface $constraint
     */
    protected function getConstraint(\TYPO3\CMS\Extbase\Persistence\QueryInterface $query, \TYPO3\CMS\Extbase\DomainObject\DomainObjectInterface $parentObject, $propertyName, $fieldValue = '', $relationTableMatchFields = array())
    {
        $columnMap = $this->getDataMap(get_class($parentObject))->getColumnMap($propertyName);
        if ($columnMap->getParentKeyFieldName() !== NULL) {
           if ($columnMap->getColumnName() == 'fal_related_files') {
                $constraint = $query->equals($columnMap->getParentKeyFieldName(), $parentObject->_getProperty('_localizedUid'));
            } else {
                $constraint = $query->equals($columnMap->getParentKeyFieldName(), $parentObject);
            }

            if ($columnMap->getParentTableFieldName() !== NULL) {
                $constraint = $query->logicalAnd($constraint, $query->equals($columnMap->getParentTableFieldName(), $this->getDataMap(get_class($parentObject))->getTableName()));
            }
        } else {
            $constraint = $query->in('uid', \TYPO3\CMS\Core\Utility\GeneralUtility::intExplode(',', $fieldValue));
        }
        if (count($relationTableMatchFields) > 0) {
            foreach ($relationTableMatchFields as $relationTableMatchFieldName => $relationTableMatchFieldValue) {
                $constraint = $query->logicalAnd($constraint, $query->equals($relationTableMatchFieldName, $relationTableMatchFieldValue));
            }
        }
        return $constraint;
    }
}

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32718

Raphael Zschorsch wrote:

in the translation of a record it should be possible to remove images or add new ones. But ONLY the images of the default record are shown in the frontend.
...but it doesn't work and only the default record images are shown.

There are many things to consider with relations and translations. We can make most use cases work in Extbase (FAL images is the most obvious one), but should also make clear that the desired behavior heavily depends on the use case and extension developers may need to adapt annotations or TCA to make it work like desired (which in turn could make it hard for backports).

If you have a simple 1:n relation (a RelationalObject has exactly one AggregateRootObject):

AggregateRootObject (ARO) -> RelationalObject (RO)

Depending on your domain, you want different behaviors of the relations in case of localization

Case 1: Relations must not change. This means, that a specific ARO is always related to a specific RO in all localizations. This means that text fields are translatable (editable), but the relation is not editable for localizations to maintain consistency.

E.g.: Product -> Brand
It makes sense to translate the product and Brand description, but it does not make sense to localize the Product/Brand as for every localization the Product will always have the same Brand.

Case 2: Relations can change for localized entities. This means AROs can have relations to different ROs for every localization. This means not only text fields are localized but also relations (id fields or intermediate tables point to localized versions of an entity)
That is certainly more complicated, as additional behavior needs to be defined. Does not found relation on localized record mean the localization has none, or should a fallback be in place?
Accessing the ARO from a RO (inverse relation) should result in the localized ARO or the default? If it is the localized ARO, how about other relations that should behave like described in case 1?

Especially case 2 can be a nightmare to think through and define what is wanted, not to speak about implementing it.
"Unfortunately" images are now (with FAL) relations to entities and what we need is exactly case 2.
This is the issue we face now and we should solve it by well defining (and documenting) how things work (or should work) under which conditions.

Oh and btw. case 2 is not implemented in Extbase.

Here is what DataHandler currently does for relation fields, when "l10n_mode" is not set to "mergeIfNotBlank" or "exclude"

AggregateRootObject (default)   ->  RelationalObject (default)

AggregateRootObject (default)   ->  RelationalObject (default)
  |-> AggregateRootObject (L1)   ->  |-> RelationalObject (L1)

What the Extbase persistence does is fetching the records in the default language and performing a language overlay (it is a bit more complicated but it does not matter for now).

The example above would then work like a charm :

Fetch AggregateRootObject (default), overlay it, Fetch relations which is RelationalObject (default) and overlay it.

This means if your TCA is correct for your main object (News TCA for fal_media is at least inconsistent), then it works if you only need to localize relations, but not have completely different relations for localized entities.

When you need the latter, then you are in case 2 hell. E.g. if we would change it to behave like proposed and always fetch images for news records based on the localized record, then we would break existing installations that relied on the fact that translated news would not need images.

So the questions are:

1. What cases are there that need to be covered
1. How should the configuration for such cases be (re-use existing TCA properties or create new ones or use annotations)
1. How can we implement something in a backwards compatible way?

Hi, Helmut.

Thank you for a detailed description!

I'm still the one, who'd like to force "Case 2" to be handled by Extbase natively for any localization (including FAL) - not by hacks, workarounds or so.
And, sure, I see the complication of possible solution and I can imagine even more overhead in persistence.

But from the other hand this is not some rarely used feature, and it becomes hard for us to explain customer, why Extbase with FAL and 6.x better, then Pibase with usual file handling and 4.x, because customer misses features, that were existing for years in old TYPO3 versions. Sure, we can go back to 'file' and 'file_reference' and write own persistance layer for other localizable RO, but then I can ask myself a same question: "why da hell I need Extbase and FAL?".

So, answering your questions:

Does not found relation on localized record mean the localization has none, or should a fallback be in place?

I think, here we can re-define a TCA option 'l10n_mode' (or add Extbase annotation?), with such options: includeNewChildren - no relation means no fallback (so, only translated and new added relations will be shown), includeNewChildrenAndFallbacks - missing relation falls back to master record (so, translated, new added and non-translated relations will be shown.

Accessing the ARO from a RO (inverse relation) should result in the localized ARO or the default? If it is the localized ARO, how about other relations that should behave like described in case 1?

Here I may be wrong, but we should have the master ARO (uid) with translated fileds (title, description, etc.), like it is already done in Extbase. But, sure, a new option in TCA (or Extbase annotation) may be added for this.

What cases are there that need to be covered?

Being able to change relations in translated versions:
1. Exchange image in relation for translated version of website.
2. Add more new images in translated version of website.

How should the configuration for such cases be (re-use existing TCA properties or create new ones or use annotations)

Good question. And it opens even more complicated topic: why do we need a TCA at all? My dreams show fututre TYPO3 version (9.x, maybe) without TCA, but with only annotations in Extbase Domain Models. As result: we don't define validation rules, pre- and post- processings twice, but use single source for 3 ways: BE forms, FE forms and internal object creation via DataMapper. In such a case, I'd prefer annotations, but in real world we can stick to new TCA properties.

How can we implement something in a backwards compatible way?

In case of new TCA properties, this can be achieved by making default properties behave, like it works now: changing relations in child records will not affect output of these records.

Hi Viktor.

Viktor Livakivskyi wrote:

I'm still the one, who'd like to force "Case 2" to be handled by Extbase natively for any localization (including FAL) - not by hacks, workarounds or so.
And, sure, I see the complication of possible solution and I can imagine even more overhead in persistence.

We absolutely need a solution.

I think, here we can re-define a TCA option 'l10n_mode' (or add Extbase annotation?)

One or the other or something different ;)
The question is: What is a good option respecting backwards compatibility, that is easy to understand and predictable in the result.

Being able to change relations in translated versions:
1. Exchange image in relation for translated version of website.
2. Add more new images in translated version of website.

Sure this is one case. But what are the consequences especially for inverse relations (as described above). It could be the case that everything turns out to work without issues, but currently I can't tell.

Any help is appreciated, especially creating extensive functional test cases. Frans started working on that:

https://review.typo3.org/#/c/32718/

I updated the change yesterday providing a more generic solution, but the place where it should be configured is still not clear.

Functional tests should check the behavior with and without the (new?) configuration and also check related things like how it works with inverse relationships.

Would be great if you could help working on that.

Hi, Helmut.

Many thanks for your feedback.

Yes, I agree with you on having good overview over possible solutions and finding best and less migration-stressy one.
For the moment I see, that re-using of TCA option 'l10n_mode' may be handy, since everyone is familiar with it and nothing needs to be changed in existing installations, which rely on current behavior.

Inverse relations also will depend on same TCA option, so it might be a case, that entity1->entity2 relation is e.g. "includeNewChildren", but entity2->entity1 realtion will be "includeNewChildrenAndFallbacks", so we can fetch master entity1, if no translation is found.

I'd like to help with functional tests, but I'm not yet familiar with their concept - we've only started to invent unit-tests in our workflow. So, I'll come back later, when I have practice with functional tests and learn more the code, that needs to be tested.

I just applied this test to a 6.2 with EXT:news. My initial findings are reported in #59192. I'm working with sys_language_mode = strict.

This change basically enables the correct use of explicitely translated or added related items.

The only thing that is missing in my case is that the l10n_mode is taken into account. If the l10n_mode is set to "mergeIfNotBlank", I would expect TYPO3 to use the related items of the default language as long as no relation in the translated version is explicitely created. As soon as there is a relation in the translated version, only the item(s) in the translated version should be considered. This is how l10n_mode "mergeIfNotBlank" works IMO.

With that patch (d39c8b2d) in TYPO3 6.2.4 and 6.2.6 images in translated news are correctly shown. But this works only for news with an parent data in default language. With news without parent data in default language no image is shown. In list and single view.

I used the last news from github. My TypoScript configuration: sys_language_mode = strict, no sys_language_overlay.

The following behavior is also very special.

German is default language and English is the first additional language in my TYPO3-Site.

• I add a new news data with image in english. In frontend the news on the english page is visible but without image.
• I add a news data with image again but now in german language. I translate this data in english and add a image (image from default langauge was not copied). This news is showing in english frontend with image. Everything is fine.
• I delete the relation between these german and english news and have now one separate german news and one separate english news. The english news has no information about his parent localization now. But in english frontend the same news is showing with image. Everything is fine. I think its the same situation like with the first news? Seeming not.
• Now i delete the german news data which was the localized parent data before. Now in english frontend the complete news is not there? Nothing is shown.

I hope anybody can understand me and can explain this.

Ok. I understand now.
I've got checked sys_file_reference. Here are my results:

1. i create a new news record in default language, then save it.
2. now i add a image to the news, change the language (1=english) of the record and save it.
3. in sys_file_reference the field sys_language_uid is filled with "0" default language but the news record is now in english

conclusion: no image in frontend

1. i create the news record and change the language to english before i saving this
2. now i go and add an image and saving the news
3. the right sys_language_uid is filled in sys_file_reference.

conclusion: image is shown in frontend

When a record in sys_file_reference is created the sys_language_uid from the last language of the record (before saving) will use and never change in future. I can change the language in the news record (and saving) but in sys_file_reference data the sys_language_uid is ever the same.

The same behavior is for sys_file_reference entries using in tt_content. But i think the frontend rendering is other. There is no problem with this.

For Extbase the sys_language_uid in sys_file_reference must be checked whenever the langugae of the referenced data is changed. Or rather checking the language of referenced data at the moment of saving.

I hope this helps to fix the problems with this patch (d39c8b2d).

Am I correct that this is also related?:

<f:link.page pageUid="{dokument.originalResource.publicUrl}">Download</f:link.page>

returns a link to the file of the default language record - instead of the one that was indicated in the localised record.

The TCA entry for this:

        'dokument' => array(
                'exclude' => 0,
                'label' => 'LLL:EXT:publikationen/Resources/Private/Language/locallang_db.xlf:tx_publikationen_domain_model_publikation.dokument',
                'config' => \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::getFileFieldTCAConfig('dokument', array(
                        'appearance' => array(
                                'createNewRelationLinkTitle' => 'LLL:EXT:cms/locallang_ttc.xlf:images.addFileReference'
                        ),
                ), $GLOBALS['TYPO3_CONF_VARS']['GFX']['imagefile_ext']),
        ),

I think so. But i don't checked this ;-). In news extension it helps to change the TCA for fal_related_files or fal_media.

https://forge.typo3.org/issues/57021

When i'm using the localize buttons in backend it's working fine. For records without default language it helps to localize the files before using.

@Karsten Hachmeister

I tried adding

                                'showSynchronizationLink' => 1,
                                'showAllLocalizationLink' => 1,
                                'showPossibleLocalizationRecords' => 1

to "appearance" in the code above, and then I was able to do some "localizing" in the backend, but none that made sense to me - and no effect in the frontend.

@Urs

I've tested again and here are my results:

1. news in default language with related file, translate this news and localizing the related file

• in frontend the file with path and title from the default language is shown
• debug {relatedFile} in fluid template and change {relatedFile.originalResource.title} -> {relatedFile.title}, then the translated tile is shown but with the path to the same file like default language. Of course that's right, only the metadata were localized. Ok, another file is needed. But when i add the new file (localized before), there is no file in frontend. You're right.

2. single translated news without a parent record in default language

• here i add a related file (localized before) and everything is fine frontend. I add a new file (localized before too) again and this is also shown in frontend

Mmmh.., crazy. In translated news it doesn't work but single news in translation works fine. The same behavior is with images.
Only for same files/media in default language and translated the localize buttons serve his functions.
When different files or media in localized news and in default language are needed, it seems to me it's required to use single news records in the relevant language.

I'm using sys_language_mode=strict and TYPO3 6.2.6

This bug is also related:
Bug #63092

It is related to others regarding wrong sys_language but this one deals with the fact that I change the language AFTER I once saved the record and had FAL/IRRE records related to it (will not update language).

Furthermore the creation of IRRE related records does only updated default language, if the language is "> 0" but this ignores all language option (-1)

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35486

I have an error in frontend if I use this patch.

#1: PHP Warning: Missing argument 4 for TYPO3\CMS\Extbase\Persistence\Generic\Mapper\DataMapper::getPreparedQuery(), called in /html/typo3/typo3_src-.2.9/typo3/sysext/extbase/Classes/Persistence/Generic/Mapper/DataMapper.php on line 332 and defined in /html/typo3/typo3_src-.2.9/typo3/sysext/extbase/Classes/Persistence/Generic/Mapper/DataMapper.php line 345

TYPO3 6.2.9

Could you please check your TYPO3-sources? That said function only has 3 arguments (typo3/sysext/extbase/Classes/Persistence/Generic/Mapper/DataMapper.php).

I added a defaultvalue to $forceTranslationOverlay in method getPreparedQuery in file typo3/sysext/extbase/Classes/Persistence/Generic/Mapper/DataMapper.php and now the patch works.

    protected function getPreparedQuery(\TYPO3\CMS\Extbase\DomainObject\DomainObjectInterface $parentObject, $propertyName, $fieldValue = '', $forceTranslationOverlay = FALSE)

With the defaultvalue for $forceTranslationOverlay patch works for me.

With the defaultvalue for $forceTranslationOverlay the patch works for me too.
Only in translated news without l18n_parent in default language I need to localize the image or files before. Otherwise they are not shown in frontend.

I made a small extension in correspondence with Stefan Frömken to override the classes of the patch as a workaround:

https://bitbucket.org/t3easy_de/faltranslation

If you extend \TYPO3\CMS\Extbase\Domain\Model\FileReference in your own extension you have to override getOriginalResource in your extended model or extend \T3easy\Faltranslation\Domain\Model\FileReference.

Thanks a lot for your extension. It helps a lot localized datasets.
Unfortunatly the image is missing if you are in not default language and the dataset is of language type -1 "All languages".

Hey,

will there be a solution in the near future? Or is there a simple workaround. I currently develop an extension where the path to the files has to be different in any language.

Patch set 2 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35486

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32718

Patch does not work for dataset of language type -1 "All languages"

Hi, I did test the extension of Jan Kiesewetter (thanks a lot!!!) and it works well in TYPO3 6.2.13-dev.

Here http://www.csthermos.it/ you can find a working example.

All those tiles in the homepage are rendered by an Extbase plugin set with Language = All.
This plugin uses one record from the model "Mosaic" set with Language = All from a storage page.
Finally this record has MM relations with all those records of model Tile.
Those tiles are in default language but are also localized in french http://www.csthermos.it/fr/
One tile "SOCIÉTÉ" has a localized image (you can see the french payoff of the logo)
I've tested also a localized tile without image and it correctly uses the default language tile image.

The only problem is that the french shows also a not localized tile "PETIZIONE IVA 22%" even if I set sys_language_overlay = hideNonTranslated (that instead works with tt_content records) but I guess that's another bug.

Another example from this website it's a product detail www.csthermos.it/prd/stufa-pelletbiomassa/sintesi-90/
if you scroll to the bottom to the DOWNLOAD section you can find a pdf catalogue in italian and switching to the french there's the french version. So it's all ok.

Tested with:
sys_language_mode = content_fallback ; 1,0
sys_language_overlay = hideNonTranslated
and
sys_language_overlay = 1

I hope that this will be fixed in 6.2 LTS because otherwise it's very very hard to handle multi-language sites that use Extbase.

Is there a chance to get the patch into core?
As Christian said, it's really hard to handle multi-language sites. Including this patch will help a lot. Even if language "all" is not supported.

Jeb, this patch is working great - besides the known issue with language "all".
But I agree with Heiko, this would be a smaller issue than the current situation.

Please integrate this fix in the core - it does not work when you use
config.tx_extbase.objects {
TYPO3\CMS\Extbase\Domain\Model\FileReference {
className = [Vendor]\[Extension]\Domain\Model\FileReference
}
}
or the new xClass-Syntax.

I spent hours before trying to change the core files directly.

For me with 6.2.14 the patch-extension https://bitbucket.org/t3easy_de/faltranslation doesn't change anything.

I am still seeing the image from the original language.

UPDATE: It worked.. sorry. Had a different problem.

This issue is so damned annoying. The Workaround extension from https://bitbucket.org/t3easy_de/faltranslation is working perfect.

Still valid in 7.3

Henrik Ziegenhain wrote:

This issue is so damned annoying. The Workaround extension from https://bitbucket.org/t3easy_de/faltranslation is working perfect.

Unfortunately this isn't a solution at all, not even a workaround tbh as l10n_mode => exclude still doesn't work with the patch.. Is there any (real) fix on the horizon as this is imho a massive problem in current TYPO3 versions/extbase.

How can it be that for almost 1.5 years there isn't any fix available?

Are there people that try to fix this problem, at least in 7.x? It's an enormous problem for multilingual website....

Thanks

Well... 6.2.15 still doesn't fix some issues..

for example:
if (is_array($data) && $data['uid'] && $data[$field]) {
$this->fileRepository = GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\Resource\\FileRepository');
$items = $this->fileRepository->findByRelation($table, $field, $data['uid']);
} else {
$items = NULL;
}

Still only returns the original language's images..

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/32718

Beware: This problem affects not only FAL references but, also TCA "select" field. If in TCA a "select"-Field is enabled in the translation, in FE extbase will show the child records from the original record.

Patch set 4 solves the problem, with one condition. in the "select" field TCA config the behaviour config has to be set.

"behaviour" => array (
  "localizationMode" => "select" 
)

thx, the extension (https://bitbucket.org/t3easy_de/faltranslation) fix the problem for me without any core patch ;)
Need to test if it breaks anything else, but it looks good now.

I asked Jan about the status of that extension. He said that it only implements the patch that already existed and that he only plans to support it until hopefully the changes can be added to the core. The extension needs a bit of work to replace/extend classes and methods (maybe just for changing one line).

@Frank: If you have it in use, could you maybe help moving the core-patch for this further?

This is my favorite bug so far.

This Error occures still in TYPO3 LTS (7.6)

Is there any solution for 7.6? Please fix this problem. Thanks a lot.

Hey guys,

I'm realy need this feature the next 2 Month für TYPO3 7.6! We develop a big platform with multilangual images and metadata. So hurry up ;)

THX

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/32718

We made a fix for TYPO3 7.6. It consists of a patched core https://github.com/dachcom-digital/TYPO3.CMS/tree/7.6.2-falfix and a hook which corrects the records in database https://github.com/dachcom-digital/typo3-faltranslations

Hopefully there will be a core fix soon

I updated https://bitbucket.org/t3easy_de/faltranslation to the current patch. Version 2 is based on master + patch set 5.

I suggest adding different configurations of sys_language_mode/sys_language_overlay and sys_file_references with sys_language_uid=-1 ([All]) to the tests.

Hi,

any news on this one? We couldn`t get l10n_mode => exclude with FAL relations running. None of the patches is working... Please help!

Sorry Jan but unluckly the version 2 of EXT:faltranslation doesn't work anymore in my use case.
If I revert it to the previous commit (ea75283ddb10f243decfc44c3350f0c48f9c0fd1) it gets back to work.

Same for the patch set 5 it only works if I re-introduce the changes of DataMapper.php removed in version 2.
Here is my final working typo3/sysext/extbase/Classes/Persistence/Generic/Mapper/DataMapper.php: http://pastebin.com/UP3nyn3P
(obviously, in this case I disabled EXT:faltranslation)

I can't find the words to say this more gently, but let's hope that some core developer will take care of this 2 years old bug cause the whole thing is getting unsustainable.

Thanks again Jan for your work!

It also doesn't work for me with the patch i dont't know exactly what i have to do ... is there a tutorial or something ? i'm also the second projekt which i have this bug so also would be happy if it would be fixed in 7.6.5 in 7.6.3 still does not work.I hope someone can help me ?
Best regard Rainer

Hello guys,

I lot of fellas like us, are waiting for some patch related with this. Can we sponsor this fix somehow? Could some Core member provide some feedback / planing regarding this issue?

Best regards,
Juan Manuel.

@Christian Toffolo
A pity to hear that v2 does not work in 7.6.
I just transferred the patch to an extension.

Would you like to submit a pull request for faltranslation to provide a temporary solution until the bug gets fixed someday*?

@Jan Schröder I don't think that a PR is needed because faltranslation already worked with TYPO3 7.6 with a checkout of this commit:
https://bitbucket.org/t3easy_de/faltranslation/commits/ea75283ddb10f243decfc44c3350f0c48f9c0fd1
Maybe it's better to make a branch for 7.6.
If you want to talk more about it please write me in Slack https://typo3.slack.com/team/ian

About your patch set 5 instead, this is my commit that works in 7.6 where I re-introduced the changes of DataMapper.php:
https://github.com/webian/TYPO3.CMS/commit/d84e7b39b6034f07d36ce7475e293d1b18c40cc6
I hope it can help.

Belated all the best to your 2nd birthday bug #57272. You survived 2 LTS releases, many codesprints and you're still here.

Sorry for the irony, but we have seen lots of features implemented since this critical blocking bug has been reported and there seems to be no investigation on this.
I know this is OSS and everybody well me can help, but the main problem seams not to be the code but the conception and communication.

So please someone inaugurated could answer some questions:

Does your stand behind the techniques Extbase and FAL + multi language?
Are there already blueprints for the conception?
Which TYPO3 version will ship a solution?
Are backports of the solution are planed for TYPO3 6.2 / 7.6?

Jan Kiesewetter wrote:

Does your stand behind the techniques Extbase and FAL + multi language?

I can't speak for the FAL guys, and me having a lot of trouble with FAL myself, can only say this:
Translations are partly broken in the core, completely in Extbase. There is noone who can care (alone) about these bugs.

About a year ago I created this project: https://github.com/alexanderschnitzler/extbase_language_tests
It's a functional test suite I created to see what is actually broken in Extbase. To this day, it received very little attention. Today I don't care any more and avoid Extbase. I am not speaking as a member of the core team but having worked on Extbase for quite some time, I can only recommend to forget about the M part of MVC in Extbase.

I have the feeling that everybody who had to to with either FAL or Extbase is happy to have left their teams some day. It's really the worst jobs in the community.

Which TYPO3 version will ship a solution?

I really hate this kind of answer but this time it's true. The version you work on. Either that, or the version you get a lot of people interested in this topic.

Are backports of the solution are planed for TYPO3 6.2 / 7.6?

Are you kidding? :D

Hi Alex,

didn't know about your functional test suite. But the related patch to this ticket was a start to do the same, with the intention to gather all use-cases. Is there a possibility to merge your stuff with the code in Gerrit so we have this in one place?

The biggest problem with the language handling is that there is not one way to go. We have the content fallback approach and the strict + some variations on this again.
There are some ideas to get this stuff solved but as you mentioned this isn't a one man job + the language/localisation handling of the core needs to be fixed first.
Please contact Benni and/or Mathias if you want/can help out to bring the localisation of core to a better state.

groet Frans

ps. https://wiki.typo3.org/Blueprints/Extbase-Translations
ps. slack channel #cig-extbase-l10n-fal

I suggest to wait 3 weeks so that we can incorporate the results of the UX week and the Waalwijk code sprint, where it is planned to put language handling in general into a broader perspective

I believe the issue is more generic. Localization in Extbase is broken for a long time. I tried to fix it here: #62921 but there can be other cases that do not work correctly.

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/32718

Helmut Hummel wrote:

I suggest to wait 3 weeks so that we can incorporate the results of the UX week and the Waalwijk code sprint, where it is planned to put language handling in general into a broader perspective

Helmut, are there any results from the UX week and the Waalwijk code sprint?

My two cents would be to bring this to T3A level and create a "taskforce" with a budget for this.

I think, one of the first steps should be to create a transparent matrix of all setting constellations and the resulting behaviour / errors.

This would have two uses:

• Integrators know what is possible and what not (instead of knocking at the extension developer's slack channel "hey why doesn't this and that work as expected") .
• Problem Zones can be detected clearly (instead of "mergeIfNotBlank not working as expected under certain circumstances")

Maybe all of this exists already – it's just my input that enough "flight altitude" might be helpful here.

Not only this task should get extra € to force fixing,

my types:

#47192 (not fixed from 3 years)
#45873 (not fixed from 3 years)

I'm sure that there are more tasks related to translations problem.

TYPO3 8 LTS is planning to release next year, realising new version with tons of new features is not serious without fixing so serious bugs with translations.
Don't get me wrong i like new features, but they are overshadowed by bugs when one of biggest client once per week reports problem with translation.

There is a taskforce doing what Urs suggested.
The thing is that we need to turn off certain, inconsistent functionality first, then we can make the system behave predictable and consistent.

I do understand that bugs are annoying, but:
You can have all the money you want: without developers having the time to fix the issue there's little we can do.
If anyone has a developer at hand willing to dig into the issue just stop by at the TYPO3 HQ in Düsseldorf, germany and we can spend as much time (and budget) on fixing the issue as we want.

• Sorting of localized records becomes wrong. But it is always wrong with TYPO3 due to the way how localization overlay is implemented.
• It does not work with "separate localization mode

• if the language id is 0, use getSysLanguageStatement method as in the link above
• if the language id is not 0, than the whole logic in extbase should change:
  • in getSysLanguageStatement:
    • load record in the localized language first
  • in doLanguageAndWorkspaceOverlay:
    • for each localized record fetch corresponding record in the default language (in
    • if such record exists, do overlay and put the result back into the array of localized records

• sorting works
• separate localization works
• l10n_mode works

• slower because there are more queries (well, there are slower things in TYPO3 anyway)
• not as described in FE localization guide (just update the guide!)

Thanks Dmitry, this sounds reasonable on a first glimpse, especially that localized records are loaded first and then missing fields filled from the default language.

Just to make sure: the localization configuration must be respected properly, no matter the implementation. Currently it is really hard to implement own backend modules for management due to the way overlays and similar are enforced.

This should work for the BE as well. There is no dependency on the FE settings or domain or TS templates in this case.

It's great to see this is being addressed, thank you very much.

What is "separate localization mode"? sys_language_mode = strict?

Perhaps this is related to this bug.

I have a tt_content element textmedia created with sys_language_uid 1.
Then I add an asset to this element.
This asset has sys_language_uid=0 in table sys_file_reference. Shouldn't this be sys_language_uid=1?

Changing the sys_language_uid manually fixes a problem with related images.

Is there a working workaround for TYPO3 7.6.9 LTS?

A workaround is to remove `l10n_mode` from `$TCA` in corresponding fields. You will have to fill them in also in translated records but at least you will have all dependencies.

Dmitry Dulepov wrote:

A workaround is to remove `l10n_mode` from `$TCA` in corresponding fields. You will have to fill them in also in translated records but at least you will have all dependencies.

You can describe it in more detail? Have you an example for this? In my extension, only the default attributes starttime and endtime have defined l10_mode.

'starttime' => array(
            'exclude' => 1,
            'l10n_mode' => 'mergeIfNotBlank',
            'label' => 'LLL:EXT:lang/locallang_general.xlf:LGL.starttime',
            'config' => array(
                'type' => 'input',
                'size' => 13,
                'max' => 20,
                'eval' => 'datetime',
                'checkbox' => 0,
                'default' => 0,
                'range' => array(
                    'lower' => mktime(0, 0, 0, date('m'), date('d'), date('Y'))
                ),
            ),
        ),

Jan Kiesewetter wrote:

I updated https://bitbucket.org/t3easy_de/faltranslation to the current patch. Version 2 is based on master + patch set 5.

This worked well for me with TYPO3 7.6.9, thanks a lot @Jan Schröder Kiesewetter!!

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/32718

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/32718

Patch set 1 for branch TYPO3_7-6 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/50003

Patch set 2 for branch TYPO3_7-6 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/50003

Jan Kiesewetter wrote:

I updated https://bitbucket.org/t3easy_de/faltranslation to the current patch. Version 2 is based on master + patch set 5.

v1.0.2 Works perfectly with TYPO3 6.2.27. Thanks guy !!

Tried now also the branch for typo3 7.x which was pushed to the review server one month ago. Same issue as in typo3 8.x - e.c. in News extension, you are unable to attach any image beside the ones from the original language - all other images are ignored. Tried different language_modes.

Hi
We've been having issues with this bug too. I've just try the patch set 8 on a Typo3 7.6.12 and the problem persist on Collections (ObjectStorage Type). On the mapResultToPropertyValue the Method $result->getFirst() is used to get the value of each child. This is later calling on the PageRepository.php the method getRecordOverlay which, when $OLmode ist configured to 'hideNonTranslated', cause the row to be unset. I don't see any easy way out of this problem, so I'm just patching the file to avoid unsetting the row if the tables is the sys_file_reference.

Maybe a solution would be to save the file_reference to be valid for all Languages (sys_language_uid = -1) when the TCA mark language inheritance. This is just an spontan idea.

Marcos Fadul wrote:

Hi
We've been having issues with this bug too. I've just try the patch set 8 on a Typo3 7.6.12 and the problem persist on Collections (ObjectStorage Type). On the mapResultToPropertyValue the Method $result->getFirst() is used to get the value of each child. This is later calling on the PageRepository.php the method getRecordOverlay which, when $OLmode ist configured to 'hideNonTranslated', cause the row to be unset. I don't see any easy way out of this problem, so I'm just patching the file to avoid unsetting the row if the tables is the sys_file_reference.

Maybe a solution would be to save the file_reference to be valid for all Languages (sys_language_uid = -1) when the TCA mark language inheritance. This is just an spontan idea.

Today i stumbled over the same problem. FileCollection and Translation doesn`t work. Can you provide your patched file with the workaround?

Thanks.
Tobias

Hello,
here you can find the code, we are using at the moment. Its comes from t3easy_de/faltranslation plus some update. It is only tested on Typo3 7.6. Feel free to use it if you like.
https://bitbucket.org/polargold/pg-fal-translation

Greetings,
Marcos

This bug does not only affect FAL references, but all mappings to database relations that use foreign keys.

Generally it would be a good idea to allow defining the selection behaviour (by uid or _localizedUid) in the domain model (i.e. via annotation). Since some domains may always want the non-translated items while others require the translated records.

Hello,

Bad luck: I've found a case that reproduces this bug on TYPO3 7.6.16.

I've installed the cb_foundation extension. That extension extends content elements with this:

$GLOBALS['TCA']['tt_content']['types']['cbfoundation_orbit']['showitem'] = "--palette--;LLL:EXT:frontend/Resources/Private/Language/locallang_ttc.xml:palette.general;general,
--palette--;LLL:EXT:frontend/Resources/Private/Language/locallang_ttc.xml:palette.header;header,
--div--;LLL:EXT:frontend/Resources/Private/Language/locallang_ttc.xml:tabs.images, image,
--div--;LLL:EXT:frontend/Resources/Private/Language/locallang_ttc.xml:tabs.appearance,
--palette--;LLL:EXT:frontend/Resources/Private/Language/locallang_ttc.xml:palette.frames;frames,
--palette--;LLL:EXT:frontend/Resources/Private/Language/locallang_ttc.xml:palette.image_settings;image_settings,
--div--;LLL:EXT:frontend/Resources/Private/Language/locallang_ttc.xml:tabs.access,
--palette--;LLL:EXT:frontend/Resources/Private/Language/locallang_ttc.xml:palette.visibility;visibility,
--palette--;LLL:EXT:frontend/Resources/Private/Language/locallang_ttc.xml:palette.access;access";


That kind of reuse from my point of view is very clever. The author is just reusing tt_content. The problem is that it links to FAL objects (images). If I translate the new content type (orbit carrousel) it gets translated, but images linked by it don't get translated.

I wanted to reuse code posted in this thread to fix that but the author created an abstract controller with this code:

@class AbstractElementController extends \TYPO3\CMS\Extbase\Mvc\Controller\ActionController {

public function showAction() {
        $data = $this->configurationManager->getContentObject()->data;
        $this->view->assign('data', $data);
    }
}@

Then the orbit controller extends the abstract controller. He did a lot with very little code. I like it very much. However this reproduces the bug in this report.

Does someone has a suggestion on how to fix this?

This is clearly an error on the core.

Still no fix for 8 LTS? ^^

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/32718

You should check out https://github.com/froemken/repair_translation, thanks to Stefan and Liddy.

Benni Mack wrote:

You should check out https://github.com/froemken/repair_translation, thanks to Stefan and Liddy.

Unfortunately it fixes the problem only for sys_file_reference. But the problem exist also for any other relation in Extbase.

Benni Mack wrote:

You should check out https://github.com/froemken/repair_translation, thanks to Stefan and Liddy.

This extension works really good. Like Benni already wrote: Thanks to Stefan and Liddy. :) But unfortunately this extension isn't working for the table pages, because the translations aren't stored in the same table in this case. But we created a patch for this case and provided a pull request for this extension. https://github.com/froemken/repair_translation/pull/1

Is this still only about FAL? Extbase has problems also with all other relations. For example, if you relate to your own table, that has the same problem.

Dmitry Dulepov wrote:

Is this still only about FAL? Extbase has problems also with all other relations. For example, if you relate to your own table, that has the same problem.

Absolutely correct, thanks for clarifying. From the start it was a problem with all relations. Extbase accounts for language in the first query but forgets about it in JOINs for dependant models.

Hey Jonas,

could you see if this patch fixes your issue?
https://review.typo3.org/#/c/53974/

could you see if this patch fixes your issue?
https://review.typo3.org/#/c/53974/

Hi Benni, yes, EDIT: I will test the patch. Sorry, messed that up.

This is for the extension solutions:
EXT:repair_translation fixes FAL-translations (as did also EXT:faltranslation). Tested with
Typo3 7.6.23
news 6.1.1, field: fal_related_files (sys_file_reference)

It does not fix
news 6.1.1, field: related (news)

• create 2 news (A+B)
  in the standard language in A:
• add a related file
• add a related news (A)
  translate A
• add another(!) related file
• add another(!) related news (B) (bonus points for choosing a translated news ;) )

-> check frontend output of news A

Without patch: standard language is fine, translated shows translated records (if they exist) of the standard language field contents (correct language, wrong records!).
The mentioned extensions fix that for sys_file_reference-fields (but not for all other relations).

The patch works in v8.7.7. I am unable to backport to v7.6.23 due to too numerous changes in \TYPO3\CMS\Extbase\Persistence\Generic\Storage\Typo3DbQueryParser.

Can confirm, EXT:repair_translation works for me with TYPO3 v8.7.12 and latest EXT:news for images.
Instantly.

Hi
The patch https://review.typo3.org/#/c/53974/ is ready for review.
Please take a moment to test it, it should also be a solution for issues mentioned here.

patch https://review.typo3.org/#/c/53974/ has been merged if I am not wrong

this issue still persists in version 8.7.x, any plans to implement the bugfix to this version as well?
At the moment installing EXT:repair_translation is the only way to fix this issue

Richard no-lastname-given wrote:

this issue still persists in version 8.7.x, any plans to implement the bugfix to this version as well?
At the moment installing EXT:repair_translation is the only way to fix this issue

Since this is a rather deep change which fundamentally changes the behavior of translations in Extbase a backport would be too risky. Also it would require additional work to e.g. backport the feature toggles and features are never backported to older versions.

So all in all, for v8 there is no good way to fix this aside from EXT:repair_translation and its variants.

EXT:repair_translation only fixes FAL translations (images and files), not all relations (e.g. related news or any other custom relation) when I last checked.

Is our current (big) patch in master really the only way to fix it? Maybe there is a smaller patch possible that can be ported?

Also I just want to note:

https://review.typo3.org/#/c/53974/84..116//COMMIT_MSG was handled as a bug fix (not feature!) and it must have slipped me that target:8.7 was dropped in patchset 85.

Hi Jonas,
Thanks for your comment.
I agree that it's a pitty the patch was not backported to v8.
As you have figured out, the patch was first targeted to be backported to older branches, and at some point where I realized how time consuming this would be I decided not to backport it.
This was a tough decision.

I estimate doing a backport for at last few days of work. This is due to the fact that in v9 few bigger changes have happened - like introduction of the context api.

It might also happen that the backport can be too risky for an stable branch, and we will not be able to ship it with the v8 core.
We don't know that before the backport is ready. Backporting the patch would also mean backporting some changes to the tests and to the testing framework.

So the bottom line is:
- backport is doable, but it is also time consuming, the new behavior can be available with feature switch as in v9 (@Mathias - it uses Extbase feature switch and not the new core API, so this doesn't block the backport)
- it's not guaranteed that it will be merged to v8, or it will be shipped as a separate extension
- the problem is the manpower

From my POV higher priority to me has work on some new features for v10 or to fix other extbase bugs for v9, than backporting this bugfix.

If many more people come to me and say that its crucial to them to have the backport, I can reconsider the priorities (and ask them to donate time or money to the project).

If somebody come up saying "I have 2 days, I would like to help with the backport" I would be more than happy to guide the person and also shift my priorities.

But without seeing engagement of others I consider other issues and features higher priority.

Please let me know if you have any more questions.

Hi Tymoteusz,

i initially bumped this issue yesterday because i encountered this bug in a recent project.

I agree with you that a backport is probably too risky, because i think many developers have already put in work-arounds in their projects to avoid this issue.
If a bugfix would be introduced in a patch release, it will probably break many existing websites.

The problem i have with this issue is that i easily spend 2 or 3 hours debugging my own code because i though it was a bug that i produced when writing my extension.
Only after i put in the work, i could find this bug entry and link it with my issue.

Isn't there any place to inform extension developers of this problem to avoid having them possibly waste their time on debugging this issue?
I think it would be important in this case because this bug regards an essential functionality of the CMS in my opinion.

AFAIK this was fixed with https://review.typo3.org/c/Packages/TYPO3.CMS/+/53974 in v9.
Please also check documentation on the new language handling in v9:

https://docs.typo3.org/c/typo3/cms-core/master/en-us/Changelog/9.5/Important-82363-MakeExtBaseTranslationHandlingConsistentWithTyposcript.html

Patch for 8.7 still under review. when will it be available?

The 8.7 patch worked for almost all cases but it appear to fail when having a record that only exists in a translation without a link to the default language.