TYPO3 Core

Impact also seen in Typo3 6.2.9 - images are getting lost in content elements.
Live Workspace is not available, it´s loading and loading, nothing happens. Click on other workspaces is possible, they are loading.

Additionally in the t3 log are several PHP Warnings, array are not closing etc. I am not sure if this belongs together but it appeared at the same time as workspace impact:

Core: Error handler (FE): PHP Warning: array_flip() expects parameter 1 to be array, null given in /home/.sites/106/site3/web/typo3_src-6.2.9/typo3/sysext/core/Classes/Charset/CharsetConverter.php line 929

Core: Error handler (FE): PHP Warning: get_html_translation_table() expects at most 2 parameters, 3 given in /home/.sites/106/site3/web/typo3_src-6.2.9/typo3/sysext/core/Classes/Charset/CharsetConverter.php line 929

Thanks for fixing.

Best regards,
Alex

I've just did a test in 6.2.14 ... same behavior.

You are able to reproduce the error with the instructions written in initial post.

Thx.
Best regards, René

Same here :(
6.2.14. with workspaces.

"router
Element "sys_file_reference:472" does not exist"

Even if a user has deleted a file reference in the backend - this error never should thrown, but ignored.

Analyzing:
UID 472 does not exists in sys_file_reference (anymore) (instead of "deleted=1")

We also got sometimes this issue. the couse is that the workspace module tries to find data by sys_reference which may fail if it isn't up to date.

Please try to run the lowlevel commands to update your sys_reference. And then try again.

Hello Alexander,

We run the cleaner_fix.sh script - same error in TYPO3 backend.

We run the "lowlevel_refindex" command - same error in TYPO3 backend.

This hint will not fix this issue.

Thx.
Best regards, René

What was the last output of the typo3/cli_dispatch.phpsh lowlevel_refindex run?

Ok, we run a development platform and a live platform.

On development platform the error 'Element "sys_file_reference:XXX" does not exist"' disappeard after running "lowlevel_refindex" command.
On live platform the error 'Element "sys_file:XXX" does not exist"' didn't disappear after running "lowlevel_refindex" command.

So there is a difference between these two errors "sys_file:XXX" and "sys_file_reference:XXX".

As I said the issue is not repeatable in older version 6.2.6 - so something was changed relating to file handling.

Thx.
Best regards, René

Till yet you spoke about the issue "sys_file_reference:XXX" which is, as you wrote, gone now. So the described fix helped you for the given issue.

Your next issue "sys_file:XXX" is another issue, but also for that, you didn't answer my last question.

Sorry about that - here are the last lines of the lowlevel_refindex run:

"Index table hosted 51 indexes for non-existing tables, now removed
251482 records from 47 tables were checked/updated.
Updates: 7337"

Thx.
Best regards, René

As the number of updates isn't zero, it seams it needs to run more times. Please take a look into the documentation for this: https://github.com/TYPO3/TYPO3.CMS/blob/master/typo3/sysext/lowlevel/README.rst

BTW: Please consider a data lose if you use Workspaces (then please do not run the orphan_records lowlevel_cleaner).

We rerun the lowlevel command. The last output was "Index Integrity was perfect!".

So, we debugged the 'Element "sys_file:XXX" does not exist"' error and found out that the following SQL query will fix this error:
"UPDATE sys_file_metadata SET pid='0' WHERE pid='-1';"

Summary:

Newer versions of TYPO3 has a problem in file handling in combination with workspaces. Last testet TYPO3 version without that issue is 6.2.6.

To fix the error 'Element "sys_file_reference:XXX" does not exist"' a "typo3/cli_dispatch.phpsh lowlevel_refindex -e" command will help.
To fix the error 'Element "sys_file:XXX" does not exist"' a "UPDATE sys_file_metadata SET pid='0' WHERE pid='-1';" SQL query will help.

Those workarounds are not applicable by customer. If somebody edit files on content elements in workspaces agin - the error will recur. Issue should be fixed in TYPO3 Core.

@Alexander: special thanks for helping out.

Best regards, René

The steps describe a way to reproduce the "sys_file_reference:XXX" issue.

We can reproduce it with a clean TYPO3 installation and our project TYPO3 instances.

Thx, René

After the steps of reproducing the error "sys_file_reference:XXX" appears in Workspace module for the "Live Workspace".

After running typo3/cli_dispatch.phpsh lowlevel_refindex -e command we do see all the things that should be displayed - not an empty site or something like that.

René

I mean, do you see elements in the Workspace module for the live Workspace? Normally you shouldn't see something there, as they are Live.

At the moment I'm waiting for reviews on https://forge.typo3.org/issues/69135 ... which may also cause this issue.

Hi,

I downloaded the latest version from git (TYPO3.CMS-cc04ac1, 75/43575/2, 6.2.16-dev), set up a clean instance and repeated the steps from the issue description above ... the issue still exists :-(

Best regards,
René

Ok, could reproduce with the mentioned steps in the workspace module in the live-workspace in 6.2.16-dev (TYPO3_6-2 branch)

Exception is thrown in ElementEntityProcessor::createNewDependentElementCallback
The missing sys_file_reference relations once was the delete-placeholder created in the edit action after the parent content element has been published the first time (according to the test-plan in the initial description of this issue). After fixing the reference index, the issue did no occur anymore.

• reference index is not purged automatically on removing a delete-placeholder upon publishing
• any far more important - the published workspace version is displayed in the workspace module of the live workspace, which should IMO not happen since it was a workspace version and is not a valid live version anymore (I'm referring to using the versioning feature without actually using workspaces)

As I mentioned above:

Only newer versions of TYPO3 has a problem in file handling in combination with workspaces. Last testet TYPO3 version without that issue is 6.2.6.

To fix the error 'Element "sys_file_reference:XXX" does not exist"' a "typo3/cli_dispatch.phpsh lowlevel_refindex -e" command will help.
To fix the error 'Element "sys_file:XXX" does not exist"' a "UPDATE sys_file_metadata SET pid='0' WHERE pid='-1';" SQL query will help.

Those workarounds ARE NOT applicable by customer. If somebody edit files on content elements in workspaces again - the error will recur. Issue should be fixed in TYPO3 Core.

None of our customer is willing to fix the reference index manually every time file references has been changed.

Its an issue of newer versions of TYPO3.

Thx, René

What are the next steps in this issue?

Best regards,
René

We ended up overwriting \TYPO3\CMS\Workspaces\ExtDirect\ActionHandler via XCLASS to cleanup the refindex table after relevant workspace actions.

ext_localconf.php:

$GLOBALS['TYPO3_CONF_VARS']['SYS']['Objects']['TYPO3\CMS\Workspaces\ExtDirect\ActionHandler'] = array(
    'className' => 'vendor\extensionname\XCLASS\ActionHandler'
);

typo3conf/ext/extensionname/Classes/XCLASS/ActionHandler.php

<?php
namespace vendor\extensionname\XCLASS;

use TYPO3\CMS\Backend\Utility\BackendUtility;
use TYPO3\CMS\Core\Utility\GeneralUtility;

/**
 * ...
 */
class ActionHandler extends \TYPO3\CMS\Workspaces\ExtDirect\ActionHandler
{

    /**
     * On all workspace actions, refresh the refindex table for affected datasets
     * ...
     */
    protected function processTcaCmd(array $cmdMapArray)
    {
        $result = parent::processTcaCmd($cmdMapArray);

        foreach ($cmdMapArray as $sTablename => $aRows) {
            if (is_array($aRows)) {
                foreach ($aRows as $iUidLive => $aVersionData) {
                    $iUidWorkspace = empty($aVersionData['version']['swapWith']) ? 0 : (int)$aVersionData['version']['swapWith'];
                    if ($iUidWorkspace) {
                        $this->updateRefIndexForWorkspaceAndLive($sTablename, $iUidLive);
                        $this->updateRefIndexForWorkspaceAndLive($sTablename, $iUidWorkspace);
                    }
                }
            }
        }

        return $result;
    }

    /**
     * If table is workspace activated, update Reference Index (sys_refindex) for a record in current and LIVE workspace
     * ...
     */
    protected function updateRefIndexForWorkspaceAndLive($table, $id)
    {
        /** @var $refIndexObj \TYPO3\CMS\Core\Database\ReferenceIndex */
        $refIndexObj = GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\Database\\ReferenceIndex');
        if (BackendUtility::isTableWorkspaceEnabled($table)) {
            $refIndexObj->setWorkspaceId(0);
            $refIndexObj->updateRefIndexTable($table, $id);

            if (!empty($GLOBALS['BE_USER']->workspace)) {
                $refIndexObj->setWorkspaceId($GLOBALS['BE_USER']->workspace);
                $refIndexObj->updateRefIndexTable($table, $id);
            }
        }
    }
}

bug also comes up in type3 7.6.2. if i delete an image (logged in user without admin-rights) in draft workspace which was already aplied to "live workspace" an error occures:
router
Element "sys_file_reference:123" does not exist

if i go to workspaces live with admin rights. if i look up in table sys_file_reference uid 123 is missing.

I have the same problem (TYPO3 6.2.12) and "typo3/cli_dispatch.phpsh lowlevel_refindex -e" could not solve it for 'Element "sys_file_reference:XXX" does not exist'. Even if I executed it multiple times.
So I deleted those entries manually to get workspaces to work again:

DELETE FROM sys_refindex WHERE ref_table = "sys_file_reference" AND ref_uid = XXX

Hey,

I tried to implement the XCLass-Solution (#33) but it did not work.
It either broke the whole typo3 with a message that it could not find the class "vendor\extensionname\XCLASS\ActionHandler" or the typo3 worked but there was no change in its behavior.
Do you have an idea if i missed something?

"typo3/cli_dispatch.phpsh lowlevel_refindex -e" works though.

Thank you for working on it.

Same issue here with TYPO3 7.6.3

We have this problem all the time on the live servers, but with tt_content and pages (i think because of templavoila), so i get a nasty command to fix this in time until we found a solution

DELETE sysref.*
FROM sys_refindex sysref
WHERE hash IN (SELECT hash
               FROM (SELECT sr.hash
                     FROM `sys_refindex` sr LEFT JOIN tt_content c ON c.uid = sr.ref_uid WHERE sr.ref_table = 'tt_content' AND c.uid is null) x);

DELETE sysref.*
FROM sys_refindex sysref
WHERE hash IN (SELECT hash
               FROM (SELECT sr.hash
                     FROM `sys_refindex` sr LEFT JOIN pages p ON p.uid = sr.ref_uid WHERE sr.ref_table = 'pages' AND p.uid is null) x);

I can reproduce this error message with TYPO3 6.2.19 doing the following:

• Switch to Draft workspace
• Upload Image1.jpg and Image2.jpg in file list
• Create new page
• Create new image element on page, select Image1.jpg, save
• Edit image element, delete Image1.jpg reference, add Image2.jpg reference, save
• Switch to file list, delete Image1.jpg (no error because it isn't referenced anymore)
• Switch to Workspace module: "sys_file_reference:XXX" does not exist"

IMHO this is a regression introduced with #64253.

Thanks to the detailed step description I was able to reproduce the issue with a functional test; please have a look at the patch set.

I'm in high doubt where to exactly put the test, but I hope you'll get the idea. Comments on how to clean this up highly appreciated.

The fix itself seems pretty straightforward, but I would love to hear if it does not interfere with the change intended by the author of #64253.

We see this in TYPO3 7.6.10

the same on 7.6.15

Same error on 7.6.16

Seems solved on 9.0.0-dev (latest master). (tested with test https://forge.typo3.org/issues/67676#note-39 )Can someone confirm?

The patch proposed triggers an error " [newlog()] Could not be edited in offline workspace in the branch where found (failure state: 'Workspace ID of record didn't match current workspace'). Auto-creation of version not allowed in workspace!" and the relation is completely lost during publishing into live-workspace (7.6.23).

Nvm - fixed as the initial relation state of these records was broken.
The patch proposed in this ticket seems to work for us on 7.6.23. Could you please integrate it into the core, so lots of users can profit from it ? I think the discussion about regression tests in this patch delaying it for 1-2 years is real unuseful.

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/47477

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/47477

I can confirm this bug still exists in 7.6.23. A fix would be very welcome.

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/47477

The bug still exists in TYPO3 7.6.24.
Error message in the draft workspace:

router
Element "sys_file_reference:2408" does not exist

sys_log-entry for this case:

INSERT INTO `sys_log` (`uid`, `pid`, `userid`, `action`, `recuid`, `tablename`, `recpid`, `error`, `details`, `tstamp`, `type`, `details_nr`, `IP`, `log_data`, `event_pid`, `workspace`, `NEWid`, `request_id`, `time_micro`, `component`, `level`, `message`, `data`) VALUES
(22010, 0, 18, 1, 2408, 'sys_file_reference', 0, 0, 'New version created of table ''%s'', uid ''%s''. UID of new version is ''%s''', 1516182473, 1, 10, '195.145.62.234', 'a:3:{i:0;s:18:"sys_file_reference";i:1;i:934;i:2;i:2408;}', 331, 1, 'NEW5a5f1bc9bc091634568134', '', 0, '', 0, NULL, NULL),
(22014, 0, 18, 2, 2408, 'sys_file_reference', 0, 0, 'Record ''%s'' (%s) was updated. (Offline version).', 1516182473, 1, 10, '195.145.62.234', 'a:2:{i:0;s:17:"hope_seitlich.jpg";i:1;s:23:"sys_file_reference:2408";}', 331, 1, '', '', 0, '', 0, NULL, NULL),
(22016, 0, 18, 2, 2408, 'sys_file_reference', 0, 0, 'Record ''%s'' (%s) was updated. (Offline version).', 1516182583, 1, 10, '195.145.62.234', 'a:2:{i:0;s:17:"hope_seitlich.jpg";i:1;s:23:"sys_file_reference:2408";}', 331, 1, '', '', 0, '', 0, NULL, NULL),
(22018, 0, 18, 2, 2408, 'sys_file_reference', 0, 0, 'Record ''%s'' (%s) was updated. (Offline version).', 1516182646, 1, 10, '195.145.62.234', 'a:2:{i:0;s:17:"hope_seitlich.jpg";i:1;s:23:"sys_file_reference:2408";}', 331, 1, '', '', 0, '', 0, NULL, NULL),
(22071, 0, 18, 0, 2408, 'sys_file_reference', 0, 0, 'Stage for record was changed to -10. Comment was: "Hopi"', 1516189151, 1, -1, '195.145.62.234', 'a:0:{}', 331, 1, '', '', 0, '', 0, NULL, NULL),
(22072, 0, 18, 6, 2408, 'sys_file_reference', 0, 0, 'Stage raised...', 1516189151, 1, 30, '195.145.62.234', 'a:2:{s:7:"comment";s:4:"Hopi";s:5:"stage";i:-10;}', -1, 1, '', '', 0, '', 0, NULL, NULL),
(22088, 0, 15, 3, 2408, 'sys_file_reference', 0, 0, 'Record ''%s'' (%s) was deleted unrecoverable from page ''%s'' (%s)', 1516192603, 1, 0, '195.145.62.234', 'a:4:{i:0;s:12:"[Kein Titel]";i:1;s:23:"sys_file_reference:2408";i:2;s:12:"[root-level]";i:3;N;}', 0, 1, '', '', 0, '', 0, NULL, NULL),

the issue persist, but maybe exists only for LIVE work space, in other workspace the issue appear to fixed
typo3 7.6.24

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/47477

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/47477

Is this problem fixed with this patch? The patch seems not to do much except updating the ref index now only if a dataset is swapped into a workspace. So updating takes less place or am I getting something wrong?

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/47477

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/47477

Patch set 9 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/47477

Patch set 10 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/47477

I repeated the test proposed at comment 39 (https://forge.typo3.org/issues/67676#note-39) on 9.5.13

Switch to Draft workspace
Upload Image1.jpg and Image2.jpg in file list
Create new page
Create new image element on page, select Image1.jpg, save
Edit image element, delete Image1.jpg reference, add Image2.jpg reference, save
Switch to file list, delete Image1.jpg (no error because it isn't referenced anymore)
Switch to Workspace module: "sys_file_reference:XXX" does not exist"

with these results:

1) No error visible on workspace module
2) there is an error in the administration log:

Core: Exception handler (WEB): Uncaught TYPO3 Exception: #1393960943: Element "sys_file_reference:53" does not exist | RuntimeException thrown in file /TYPO3-dists/typo3_src-9.5.13/typo3/sysext/workspaces/Classes/Dependency/ElementEntityProcessor.php in line 187. Requested URL: https://typo3.9.test.it:8890/typo3/index.php?route=%%2Fajax%%2Fworkspace%%2Fdispatch&token=--AnonymizedToken--

3) The workspace module does not list the page nor the content element (I expected to see them) so it seems somehow broken. If I remove the "image2.jpg" reference,too then both the page and the content element are now listed in the workspace module.

Patch set 11 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/47477

Patch set 12 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/47477

Patch set 13 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/47477

Patch set 14 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/47477

Re-test the scenario from https://forge.typo3.org/issues/67676#note-64

-> Still broken: the workspace module ajax call that fetches content breaks with #1393960943 RuntimeException "Element "sys_file_reference:1869" does not exist" and it is logged as ERROR in log module.

Simplified scenario:

Have a clean refindex.

• Go to workspace
• Upload a fresh image in filelist module
• Add a page
• Add a content element, add the image to the CE, save
• Delete the image in CE again, save
• Go to filelist, delete the image

> workspace modul ajax call throws exeption
> calling "bin/typo3 referenceindex:update" fixes the situation

Technical background:

• 2 rows in sys_file_reference are created (state 1 & 1) for the tt_content>sys_file relation (correct)
• An entry in sys_refindex is created from tt_content element to sys_file_reference

• The two rows in sys_file_reference are set to deleted=1 (correct)
• The sys_refindex row is NOT set to deleted=1 (not correct)

-> there is a dangling sys_refindex entry now.

• sys_file has no soft delete behavior, so the sys_file row is removed from DB
• The two rows in sys_file_reference are removed from db
• The sys_refindex row is NOT touched

-> The sys_refindex row now points to no longer existing record. This triggers the exception.

---> There is a missing updateRefIndex() call somewhere in the DataHandler delete operation.

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65809

Patch set 1 for branch 10.4 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/65827