Bug #75399

Extbase: Class/Table Mapping doesn't work sometimes

Added by Christian Huppert about 4 years ago. Updated 1 day ago.

Status:
Accepted
Priority:
Must have
Assignee:
Category:
Extbase
Target version:
-
Start date:
2016-04-04
Due date:
% Done:

0%

TYPO3 Version:
9
PHP Version:
7.2
Tags:
Complexity:
Is Regression:
No
Sprint Focus:

Description

TYPO3: 6.2.19
PHP-Version: 5.4.x
SQL-Version: 5.5.x
Powermail-Version: 2.17.1 (last version working with 5.4.x)

Hi,

sometimes it happens on our Production instance that a page where a Powermail form is located is not visible as the "Oops an error occured" page is shown up.

If you investigate the corresponding issue on https://forge.typo3.org/issues/70024 you will find out that the cause for this is that Extbase did not correctly map the tables
like setup in config.tx_extbase of the powermail extension.

If one deletes all caches then the mapping is correct and the page displays properly again until after some time (hours or days or weeks) in future the error reappears again.

Also it seems that the error only appears in Production mode. I never had it on my dev environment.

Please do not think that it is a powermail issue. It is an issue of the Extbase class/table mapping procedure.

Maybe this could be also of interest to you:

Stefan Gruber says in the forge ticket above the following:

I could reproduce the issue if this fails in DataMapFactory.php*

94: $frameworkConfiguration = $this->configurationManager->getConfiguration(\TYPO3\CMS\Extbase\Configuration\ConfigurationManagerInterface::CONFIGURATION_TYPE_FRAMEWORK);

Definitely not a powermail issue!

* He means: ./typo3/sysext/extbase/Classes/Persistence/Mapper/DataMapFactory.php

Could you please take some time to investigate this issue?

This would be great :)
Thank you.

Cheers
Christian

DataMapFactory.php View (20.9 KB) Sander Leeuwesteijn, 2018-05-03 13:22

cf_extbase_datamapfactory_datamap-content.wrong.bin (788 Bytes) Markus Mächler, 2020-06-05 12:29

cf_extbase_datamapfactory_datamap-content.correct.bin (15.8 KB) Markus Mächler, 2020-06-05 12:29


Related issues

Related to TYPO3 Core - Bug #78091: cf_extbase_datamapfactory_datamap Enties are generated wrong after expiring Closed 2016-09-28
Related to TYPO3 Core - Bug #83232: Table 'tx_extbase_domain_model_filereference' doesn't exist in \/var\/www\/live.ejw-manager.de\/typo3_src-8.7.8\/typo3\/sysext\/extbase\/Classes\/Persistence\/Generic\/Storage\/Typo3DbBackend.php:384 Closed 2017-12-05
Duplicated by TYPO3 Core - Bug #77321: Similar Behaviour of mapping error as described in Bug #66952 Closed 2016-08-01
Duplicated by TYPO3 Core - Bug #78292: TYPO3 6.2.27 creates wrong cache files Rejected 2016-10-14

History

#1 Updated by Christian Huppert about 4 years ago

So today this behaviour arose again. The last time it arose was 1st April 2016.

#2 Updated by Christian Huppert about 4 years ago

I found out that the problem lies in the fact that under not yet found conditions the expire-date in the table "cf_extbase_datamapfactory_datamap" is not updated and if the expiration date is reached the config.tx_extbase for the corresponding identifier is not delivered and hence the extension cannot work and hence the "Oops an error occured" message shows up.

#3 Updated by Muriel le Pair over 3 years ago

TYPO3 6.2.27
PHP 5.4.45

Same issue here.

I have updated a multidomain configuration to TYPO3 6.2.27.
Sometimes the cache files are created incorrect, resulting in a nasty error (the odd thing is that some sites will work and others will produce the following error (all sites use the extension):

Oops, an error occurred!
Table '[DB].tx_sfaccordion_domain_model_ttcontent' doesn't exist
refering to: https://wiki.typo3.org/Exception/CMS/1247602160

The mapping seems to be correct:

#sf: added
    persistence {
        classes {
            SF\SfAccordion\Domain\Model\TtContent {
                subclasses {
                    itemrows = SF\SfAccordion\Domain\Model\Plugins\ItemRows
                }
            }
            SF\SfAccordion\Domain\Model\Plugins\ItemRows {
                mapping {
                    #ctype
                    recordType = sfaccordion_pi1 
                    tableName = tt_content
                    columns {
                        tx_sfaccordion_content.mapOnProperty = rows
                    }
                }
            }
        }
    }

When I go to 'Install' > click on 'Remove all cache', the site works again. But then all of a sudden the error reappears.
So it looks like sometimes the cache is created wrong.

#4 Updated by Muriel le Pair over 3 years ago

I have transfered the site to a server with PHP 5.6.16, the same error occurs. So the bug is not related to the PHP 5.4.
The error seems to occur randomly on a daily base, but mostly in the morning.

#5 Updated by Muriel le Pair over 3 years ago

I can confirm that this bug is releted to https://forge.typo3.org/issues/78091.

The first time the content is created in cf_extbase_datamapfactory_datamap it works fine, the content size is 57,7 KiB.
After about 24 hours the cache expires and the content is recreated, it now has a size of 735 B and non of the sites work anymore until the cache is deleted from the install tool.
Doing so will empty the table, and when the cache is created for the first time all works fine.

#6 Updated by Muriel le Pair over 3 years ago

I found the cause and a temporary solution.

I have a multiple site setup:
- site 1
- site 2
- site 3
- site 4

Site 1 uses extensions A,C, site 2 uses extensions A,B,C,D, site 3 uses extension A,E,F,G etc etc.
When the cache expires and the first site is called before the others, this will trigger the error.

The sollution is to add all extbase extensions to the first site. Not nessasery to do with all the others.

#7 Updated by Artur Cichosz almost 3 years ago

We experienced the same problem described in #6 in a multisite project.

In our case we used a basic EXT:news on site A, but on site B we used an extension to the news called eg. EXT:some_news_extended, which extended some base news subclasses.
So the extbase property map for eg. the model GeorgRinger\News\Domain\Model\News would be different for both sites because we extended a subclass GeorgRinger\News\Domain\Model\NewsDefault.
Now if we hit site A first, its mapping will be cached under the identifier GeorgRinger\News\Domain\Model\News.
Site B looks for the model GeorgRinger\News\Domain\Model\News as well and it gets the cached version from site A which references the wrong subclass (base GeorgRinger\News\Domain\Model\NewsDefault) which in turn lacks all the new properties from our extended subclass.

Two sollutions came in my mind after I discovered what's going on:

1) Let the PropertyMapFactory create distinct cache identifiers according to the active site tree we are on in frontend. But what about the backend? Will it cause follow up issues there?

2) Disable property map cache by $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['extbase_datamapfactory_datamap']['backend'] = 'TYPO3\CMS\Core\Cache\Backend\NullBackend'. This will slow down the entire system of course, but it is a quick workaround to make the sites work properly.

I will go now for solution 2) mybe somebody can give some hints how to achieve 1)

#8 Updated by Susanne Moog over 2 years ago

  • Category set to Extbase

#9 Updated by Daniel Haupt over 2 years ago

We are experiencing the same issue with Typo3 8 LTS and PHP 7.2.

#10 Updated by Alexander Opitz about 2 years ago

  • Related to Bug #83232: Table 'tx_extbase_domain_model_filereference' doesn't exist in \/var\/www\/live.ejw-manager.de\/typo3_src-8.7.8\/typo3\/sysext\/extbase\/Classes\/Persistence\/Generic\/Storage\/Typo3DbBackend.php:384 added

#11 Updated by Jan Radecker about 2 years ago

  • TYPO3 Version changed from 6.2 to 8
  • PHP Version changed from 5.4 to 7.0

We have the same problem with TYPO3 8.7 and Php 7.0.
Sometimes it loses the mappig configuration for the extended fe_user table.

#12 Updated by Sander Leeuwesteijn about 2 years ago

We are experiencing the same issue on 6.2.31! (PHP 5.6.35)

We also have an extended fe_user model (with mapping in TS) and sometimes when the cache entry expires for this model in cf_extbase_datamapfactory_datamap the entry BLOB is generated much smaller. And the object only contains the uid. (stuff like username etc is missing)

This causes the site to malfunction and obviously stuff like logging in with an fe_user does not work.

We are still looking for the root cause. (We were using igbinary for serialization and are moving back to regular php serialization, but not sure if this is the root cause)

#13 Updated by Sander Leeuwesteijn about 2 years ago

It was not caused by igbinary.

Further debugging shows that when this happens:

sysext/extbase/Classes/Persistence/Generic/Mapper/DataMapFactory.php method buildDataMapInternal around line 94:

$frameworkConfiguration = $this->configurationManager->getConfiguration(\TYPO3\CMS\Extbase\Configuration\ConfigurationManagerInterface::CONFIGURATION_TYPE_FRAMEWORK);

returns only a small, almost empty array:

Array
(
    [persistence] => Array
        (
            [storagePid] => 0
        )

    [controllerConfiguration] => Array
        (
        )

)

which obviously is wrong, it should contain the mapping information of many extensions.

Trying to find why this happens..

#14 Updated by Sander Leeuwesteijn about 2 years ago

I don't have a real way to reproduce this and run it through the debugger (would probably find it very quickly then)

But i do have a way of running uncached on production which causes the error alot, and i added some logging to the DataMapFactory, getting closer.

Just found that this can only happen if typoscript isnt available, and when it happens, the value of $GLOBALS['TSFE']->tmpl->setup is only:

Array
(
    [styles.] => Array
        (
            [insertContent] => CONTENT
            [insertContent.] => Array
                (
                    [table] => tt_content
                    [select.] => Array
                        (
                            [orderBy] => sorting
                            [where] => colPos=0
                            [languageField] => sys_language_uid
                        )

                )

        )

    [config.] => Array
        (
            [extTarget] => _top
            [uniqueLinkVars] => 1
        )

)

This should be fetched in: sysext/extbase/Classes/Configuration/AbstractConfigurationManager.php line 191:

$setup = $this->getTypoScriptSetup();

Trying to find why it's not available now..

#15 Updated by Sander Leeuwesteijn about 2 years ago

I have found the source of these problems in this particular website. It has many custom made extensions. It occured in 2 cases:

1) In one extension the DataMapper was manually called, providing the class name with a leading \ in front of it. Which is not how it appears in the TSFE array of typoscript, the key name there is without leading \.

2) When an extension manually creates a new TSFE object, make sure it is a complete one. There was an extension here which created a new TypoScriptFrontendController and manually called some methods like:

$GLOBALS['TSFE']->initFEuser();
$GLOBALS['TSFE']->determineId();
$GLOBALS['TSFE']->initTemplate();

How ever it did not call $GLOBALS['TSFE']->getConfigArray(); so there was no TS. When you have extensions which manually build a new TSFE environment, (for example solr does this), make sure all the calls are done and the TSFE is a complete one before you call any Extbase methods like findByUid.

It does remain strange why on some occasions it does work fine and on others it does not. But i have installed some logging in sysext/extbase/Classes/Persistence/Generic/Mapper/DataMapFactory.php and the problem does not seem to occur anymore. For now ;)

I have attached the modified core file with the logging, should anyone be interested. See the // LOGGING START comments.

#16 Updated by Sander Leeuwesteijn about 2 years ago

Found another cause, watch out for core hooks which are too early in the rendering process, mainly before $TSFE->getConfigArray(); in index_ts.php.

If you use hooks before that and use Extbase style queries in them, TS is not loaded yet and the Extbase mapper will fail.

#17 Updated by Sander Leeuwesteijn about 2 years ago

Last update, the best place to find how this is happening is adding logging in sysext/extbase/Classes/Configuration/AbstractConfigurationManager.php, modify method getExtbaseConfiguration() to be something like: (and add a log method ofc)

/**
     * Returns the TypoScript configuration found in config.tx_extbase
     *
     * @return array
     */
    protected function getExtbaseConfiguration() {
        $setup = $this->getTypoScriptSetup();
        $extbaseConfiguration = array();
        if (isset($setup['config.']['tx_extbase.'])) {
            $extbaseConfiguration = $this->typoScriptService->convertTypoScriptArrayToPlainArray($setup['config.']['tx_extbase.']);
        } else {
            $this->log('config tx_extbase. was empty!!'.PHP_EOL.print_r(debug_backtrace(2), true));
        }
        return $extbaseConfiguration;
    }

public function log($msg) {
        @file_put_contents(PATH_site . 'typo3log/errors/'.date('Ym').'_ConfigManager.log', '['.date('d-m-y H:i:s').'] - '.$GLOBALS['UNIQUE_REQUEST_ID'].' - '.$msg.PHP_EOL, FILE_APPEND);
    }

With the backtrace generated there you should be able to pinpoint where it's coming from.

To generate more hits on your site to reproduce it, disable the cache for the mapper first:

$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['extbase_datamapfactory_datamap']['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\NullBackend';

#18 Updated by Steffen Wargalla almost 2 years ago

Thank you so Sander for this excellent bug tracking! This helps a lot. Much appreciated!
I was able to reproduce this issue and confirm that the real cause of this is the empty Typoscript template.

#19 Updated by Frank Krüger 7 months ago

  • TYPO3 Version changed from 8 to 9
  • PHP Version changed from 7.0 to 7.2

We have the same issues with TYPO3 8 and 9 LTS, especially when using mapped tables via TS (persistence.classes.<model>.mapping.tableName).

#20 Updated by Jan Kornblum 7 months ago

Same issues here with mapped tables (fe_users in this case). We try to restore domain objects after serialization by implementing \Serializable interface to the outer class. On unserialization, the objects are fetched back from repository, which works fine for regular (not mapped) objects. For mapped objects instead, the typoscript mapping isn't respected.

class Test implements \Serializable {

    public function unserialize($serialized) {
        $unserialized = unserialize($serialized);
        $objectManager = GeneralUtility::makeInstance(ObjectManager::class);
        // Works fine...
        $this->project = $objectManager->get(ProjectRepository::class)->findByUid($unserialized['project']);
        // Fails, tries to ask db.tx_myext_domain_model_employee instead of db.fe_users
        $this->employee = $objectManager->get(EmployeeRepository::class)->findByUid($unserialized['employee']);;
    }

}

#21 Updated by Matthias Toscanelli 6 months ago

I have the same problem with one of my sites. I have some calls made via EID.
The script the "DataMapper" to map the objects.
If a call to the script is made before the mapping cache is generated, then it corrupts the cache.
The reason is that the TypoScript of the framework is not loaded and therefore it cannot generate the mapping.

The issue is in DataMapFactory in method buildDataMapInternal (line 114).
We could perhaps detect that the framework is not loaded before calling a mapping and throw an exception so developers will be forced to make sur that the framework is loaded before calling a script that require data mapping.

To reproduce the problem:
  • This will work only with entity that use a specific mapping (not the standard tx_myextension_domain_model_myentity), because by default DataMapFactory will use this name.
  • Create a category (you need at least an entity)
  • Create a script that load this category from database
    $objectManager = GeneralUtility::makeInstance(ObjectManager::class);
    $objectManager->get(CategoryRepository::class)->findAll();
    
  • Configure "EID" to load this script or any kind of alternative that don't load TSFE configuration
  • Clear all the caches
  • Call the script
  • Call a page that require also the "Category" entity, and you will get "Table 'tx_extbase_domain_model_category' doesn't exist

Matthias

#22 Updated by Alexander Vogt 5 months ago

I have a similar problem with a middleware that uses extbase models. My middlware sometimes throws exceptions because of missing extbase mapping configurations. The error occurs when a cache entry in the cache "cf_extbase_datamapfactory_datamap" is deleted by the redis garbage collector (or something else). In order to fix the error, all caches have to be cleared.

Steps to reproduce
  • Create a middleware with configuration "after: typo3/cms-frontend/page-argument-validator" to ensure tsfe loading
  • Add a extbase query that relies on a mapping typoscript configuration to the middleware. Example config: config.tx_extbase.persistence.classes.Demo\Domain\Model\MyModel.mapping.tableName = demotablename
  • Call the middleware in the frontend: Everything works.
  • Clear the mapping cache (cf_extbase_datamapfactory_datamap) with the identifier "Demo\Domain\Model\MyModel" manually (to simulate the redis garbage collector)
  • Call the middleware in the frontend: In this case the TypoScript setup ($GLOBALS['TSFE']>tmpl>setup) is empty and the missing datamap entry will be recreated without the typoscript mapping (DataMapFactory->buildDataMapInternal) and this leads to a extbase mapping error. To fix the error, all caches have to be cleared.

#23 Updated by Paul Beck 3 months ago

I suspect this behavior will also be caused by alot of custom Routing-Aspects written for extensions Slugs. If match id2alias or alias2id and use Extbase repositories for those functions, it may also blows up your Datamap as described middleware problem from Alexander Vogt.

#24 Updated by Alexander Grein 3 months ago

I had realized the same problem after updating a page from 8.7 (using realurl) to 9.5 (using typo3 route enhancer).

After this update the speaking link generation of ext:calendarize fails after a while, because of a missing table.
The cause is, that I used an extended event model with a table mapping defined in ext_typoscript_setup.txt.

The correct mapping gets lost after a while (maybe if the datamapfactory cache expires?) and will be generated wrong again.

As far as I understand the code, if the DatamapFactory::buildDataMap didn't find a (valid) datamap, it calls the DatamapFactory::buildDataMapInternal method.

Because of the already by Sander Leeuwesteijn descripted problem (see https://forge.typo3.org/issues/75399#note-13), since the following call inside DatamapFactory::buildDataMapInternal

$frameworkConfiguration = $this->configurationManager->getConfiguration(\TYPO3\CMS\Extbase\Configuration\ConfigurationManagerInterface::CONFIGURATION_TYPE_FRAMEWORK);

only returns a small, almost empty array, the tableName gets not overriden by the mapping definition and subsequently all slug generations fails with an exception.

Meanwhile I thought I found a workaround, but it's not working :-(

My idea was to add the missing template service by changing the TYPO3\CMS\Extbase\Configuration\FrontendConfigurationManager::getTypoScriptSetup() method to the following:

    /**
     * Returns TypoScript Setup array from current Environment.
     *
     * @return array the raw TypoScript setup
     */
    public function getTypoScriptSetup()
    {
        if (!$GLOBALS['TSFE']->tmpl instanceof TemplateService) {
            // template service is not yet initialized -> do it
            $GLOBALS['TSFE']->tmpl = $this->objectManager->get(TemplateService::class);
            $GLOBALS['TSFE']->tmpl->generateConfig();
        }

        return $GLOBALS['TSFE']->tmpl->setup;
    }

Unfortunately this would fix this problem, but produce others in the end.

After digging deeper into this, I think the main reason for all of this problems is, that the TYPO3\CMS\Extbase\Middleware\TypoScriptFrontendInitialization::process(), which should initialize the typoscript frontend object, did not get any params over GPs if its called by the slug resolving routine:

$GLOBALS['TSFE'] = GeneralUtility::makeInstance(
            TypoScriptFrontendController::class,
            null,
            GeneralUtility::_GP('id'),
            GeneralUtility::_GP('type'),
            null,
            GeneralUtility::_GP('cHash'),
            null,
            GeneralUtility::_GP('MP')
        );

Because of this missing information or the fact that the TYPO3\CMS\Extbase\Middleware\PrepareTypoScriptFrontendRendering::process() will never be reached in this case, the typoscript frontend controller is not fully initialized.
But it has to be, otherwise the later call
$frameworkConfiguration = $this->configurationManager->getConfiguration(\TYPO3\CMS\Extbase\Configuration\ConfigurationManagerInterface::CONFIGURATION_TYPE_FRAMEWORK);

will not work as aspected.

This all is a chicken or the egg problem.

#25 Updated by Alexander Schnitzler 3 months ago

  • Assignee set to Alexander Schnitzler

#26 Updated by Ralf Hettinger about 2 months ago

Further debugging leads to the following scenario for me with an explanation why it actually might fail and a workaround:

1 Instantiation of TSFE

TSFE instantiated $cached (with $no_cache = false):

$tsFe = GeneralUtility::makeInstance(
  TypoScriptFrontendController::class, 
  null,
  0,
  0,
  $no_cache = false
);

2 Cache hit

Cache hit ($tsFe->config is set)

$tsFe->getFromCache();

Afterwards, the config array is built:

$tsFe->getConfigArray();

Since there has been a cache hit before, $tsFe->config is already set by $tsFe->getFromCache(). Therefore $tsFe->tmpl->start() is not called by $tsFe->getConfigArray() and thus $GLOBALS['TSFE']->tmpl->setup of TemplateService is not set (even though it could have been set by the cached data).
Imo, the initial thought was: if TSFE is coming from Cache (cache hit), then there is no need for TypoScript to be set within the TemplateService.

3 Extbase

Extbase currently needs the TypoScript via TemplateService; a call against Extbase's getTypoScriptSetup may return an empty TS template:

    /**
     * Returns TypoScript Setup array from current Environment.
     *
     * @return array the raw TypoScript setup
     */
    public function getTypoScriptSetup()
    {
        return $GLOBALS['TSFE']->tmpl->setup;
    }

4 Workaround

This workaround works for me:

    /**
     * Returns TypoScript Setup array from current Environment.
     *
     * @return array the raw TypoScript setup
     */
    public function getTypoScriptSetup()
    {
        $tsFe = &$GLOBALS['TSFE'];
        // TypoScript template was not marked as loaded
        if (!$tsFe->tmpl->loaded) {
            // try to load the TypoScript template
            $tsFe->tmpl->start($tsFe->rootLine);
        }
        return $tsFe->tmpl->setup;
    }

#27 Updated by Johannes Seipelt 17 days ago

I have the same issue as described (properties of subclasses emtpy ~1 hour after cache cleared) in two projects I upgraded from 8 LTS to 9 LTS. Current workaround for 9:

https://github.com/n3amil/extbase/commit/c1085101b0d4c80a079e413df82428c69c6b17ec.patch

The bug also occurred in 8, workaround is the same.

For 8:

Index: typo3/sysext/extbase/Classes/Persistence/Generic/Mapper/DataMapFactory.php
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- typo3/sysext/extbase/Classes/Persistence/Generic/Mapper/DataMapFactory.php    (date 1513872003482)
+++ typo3/sysext/extbase/Classes/Persistence/Generic/Mapper/DataMapFactory.php    (date 1513871962000)
@@ -240,6 +240,7 @@
      */
     protected function getColumnsDefinition($tableName)
     {
+        \TYPO3\CMS\Core\Utility\ExtensionManagementUtility::loadExtTables();
         return is_array($GLOBALS['TCA'][$tableName]['columns']) ? $GLOBALS['TCA'][$tableName]['columns'] : [];
     }

#28 Updated by Markus Klein 17 days ago

  • Assignee changed from Alexander Schnitzler to Markus Klein

@Ralf I think your workaround is a feasible solution. Would you be so kind as to push a patch to Gerrit?

#29 Updated by Markus Klein 17 days ago

  • Status changed from New to Accepted

#30 Updated by Markus Mächler 1 day ago

We are encountering the same issue when trying to use a Repository in a Middleware with TYPO3 9. Whenever the cache is empty and the middleware is called before any regular page hit, the cache entry is wrong. Afterwards regular pages do not work either. If however a regular page is called first, the cache is correct and also our middleware works.

The fix from @Ralf did not work for us. We worked around this issue by not using the Extbase ORM in our middleware but fetching the records using the QueryBuilder.

#31 Updated by Markus Klein 1 day ago

@Markus: Can you precise which cache is exactly causing the issue?

Whenever the cache is empty and the middleware is called before any regular page hit, the cache entry is wrong.

You mean that "there is no cache entry" not "cache entry is wrong", right? (caches are empty you say as first part of the sentence)

#32 Updated by Markus Mächler 1 day ago

@Markus No, to me it really looks like the cache entry for TYPO3\CMS\Extbase\Domain\Model\FileReference in cf_extbase_datamapfactory_datamap is wrong.

If the cache is empty and I call the middleware first, I get the cache entry in cf_extbase_datamapfactory_datamap-content.wrong.bin. Afterwards regular pages (probably only when using an Extbase file reference) do not work.

If however I call a regular page first, I get the cache entry in cf_extbase_datamapfactory_datamap-content.correct.bin.
Afterwards all the pages and also the middleware do work.

#33 Updated by Markus Klein 1 day ago

Thanks for clarification. Okay so we simply have missing data (as denoted above already by others) when constructing the datamap, which is cached then.

Also available in: Atom PDF