Bug #67676

Element "sys_file_reference:XXX" does not exist"

Added by Rene Sonntag over 3 years ago. Updated 3 months ago.

Status:
Under Review
Priority:
Must have
Assignee:
Category:
Workspaces
Target version:
-
Start date:
2015-06-22
Due date:
% Done:

0%

TYPO3 Version:
7
PHP Version:
Tags:
Complexity:
medium
Is Regression:
No
Sprint Focus:

Description

Error message:
"router
Element "sys_file_reference:XXX" does not exist"

How to reproduce:

- install a new clean TYPO3 6.2.13
- activate workspaces extension
- create a DRAFT workspace
- in LIVE workspace:
- create a new page
- create a content element "Text & Images"
- add an image
- goto DRAFT workspace:
- copy the content element
- paste the content element
- activate the new copied content element
- publish to live
- edit the new copied element
- add a new image
- delete the old image
- publish to live
- goto Workspaces - LIVE workspace ... the error message will appear

Testet in TYPO3 6.2.11, 6.2.12, 6.2.13
Not repeatable in TYPO3 6.2.6

Thanks for fixing.

Best regards,
René

ss_reference.PNG View (66.3 KB) Rene Sonntag, 2015-06-22 12:55

ss_typo_workspace.PNG View (54 KB) Rene Sonntag, 2015-09-07 09:58

fehler_entwurfsumgebung.png View - error message (7.72 KB) Kurt Gusbeth, 2018-02-15 16:39


Related issues

Related to TYPO3 Core - Bug #67533: Legacy uploadfolder does not work correct in workspaces Closed 2015-06-17
Related to TYPO3 Core - Bug #73062: fal images in workspaces don't work in extension New 2016-02-01

History

#1 Updated by Alex Senge over 3 years ago

Impact also seen in Typo3 6.2.9 - images are getting lost in content elements.
Live Workspace is not available, it´s loading and loading, nothing happens. Click on other workspaces is possible, they are loading.

Additionally in the t3 log are several PHP Warnings, array are not closing etc. I am not sure if this belongs together but it appeared at the same time as workspace impact:

Core: Error handler (FE): PHP Warning: array_flip() expects parameter 1 to be array, null given in /home/.sites/106/site3/web/typo3_src-6.2.9/typo3/sysext/core/Classes/Charset/CharsetConverter.php line 929

Core: Error handler (FE): PHP Warning: get_html_translation_table() expects at most 2 parameters, 3 given in /home/.sites/106/site3/web/typo3_src-6.2.9/typo3/sysext/core/Classes/Charset/CharsetConverter.php line 929

Thanks for fixing.

Best regards,
Alex

#2 Updated by Wouter Wolters over 3 years ago

  • Status changed from New to Needs Feedback

Can you try a more recent version of 6.2?

Between 6.2.9 and 6.2.14 there are several fixes included for workspaces. This problem should be solved now.

#3 Updated by Rene Sonntag over 3 years ago

I've just did a test in 6.2.14 ... same behavior.

You are able to reproduce the error with the instructions written in initial post.

Thx.
Best regards, René

#4 Updated by Stefan Maischner over 3 years ago

Same here :(
6.2.14. with workspaces.

"router
Element "sys_file_reference:472" does not exist"

Even if a user has deleted a file reference in the backend - this error never should thrown, but ignored.

Analyzing:
UID 472 does not exists in sys_file_reference (anymore) (instead of "deleted=1")

#5 Updated by Alexander Opitz over 3 years ago

We also got sometimes this issue. the couse is that the workspace module tries to find data by sys_reference which may fail if it isn't up to date.

Please try to run the lowlevel commands to update your sys_reference. And then try again.

#6 Updated by Rene Sonntag over 3 years ago

Hello Alexander,

We run the cleaner_fix.sh script - same error in TYPO3 backend.

We run the "lowlevel_refindex" command - same error in TYPO3 backend.

This hint will not fix this issue.

Thx.
Best regards, René

#7 Updated by Alexander Opitz over 3 years ago

What was the last output of the typo3/cli_dispatch.phpsh lowlevel_refindex run?

#8 Updated by Rene Sonntag over 3 years ago

Ok, we run a development platform and a live platform.

On development platform the error 'Element "sys_file_reference:XXX" does not exist"' disappeard after running "lowlevel_refindex" command.
On live platform the error 'Element "sys_file:XXX" does not exist"' didn't disappear after running "lowlevel_refindex" command.

So there is a difference between these two errors "sys_file:XXX" and "sys_file_reference:XXX".

As I said the issue is not repeatable in older version 6.2.6 - so something was changed relating to file handling.

Thx.
Best regards, René

#9 Updated by Alexander Opitz over 3 years ago

Till yet you spoke about the issue "sys_file_reference:XXX" which is, as you wrote, gone now. So the described fix helped you for the given issue.

Your next issue "sys_file:XXX" is another issue, but also for that, you didn't answer my last question.

#10 Updated by Rene Sonntag over 3 years ago

Sorry about that - here are the last lines of the lowlevel_refindex run:

"Index table hosted 51 indexes for non-existing tables, now removed
251482 records from 47 tables were checked/updated.
Updates: 7337"

Thx.
Best regards, René

#11 Updated by Alexander Opitz over 3 years ago

As the number of updates isn't zero, it seams it needs to run more times. Please take a look into the documentation for this: https://github.com/TYPO3/TYPO3.CMS/blob/master/typo3/sysext/lowlevel/README.rst

BTW: Please consider a data lose if you use Workspaces (then please do not run the orphan_records lowlevel_cleaner).

#12 Updated by Rene Sonntag over 3 years ago

We rerun the lowlevel command. The last output was "Index Integrity was perfect!".

So, we debugged the 'Element "sys_file:XXX" does not exist"' error and found out that the following SQL query will fix this error:
"UPDATE sys_file_metadata SET pid='0' WHERE pid='-1';"

Summary:

Newer versions of TYPO3 has a problem in file handling in combination with workspaces. Last testet TYPO3 version without that issue is 6.2.6.

To fix the error 'Element "sys_file_reference:XXX" does not exist"' a "typo3/cli_dispatch.phpsh lowlevel_refindex -e" command will help.
To fix the error 'Element "sys_file:XXX" does not exist"' a "UPDATE sys_file_metadata SET pid='0' WHERE pid='-1';" SQL query will help.

Those workarounds are not applicable by customer. If somebody edit files on content elements in workspaces agin - the error will recur. Issue should be fixed in TYPO3 Core.

@Alexander: special thanks for helping out.

Best regards, René

#13 Updated by Alexander Opitz over 3 years ago

  • Category set to Workspaces

Hmmm, this sounds as sys_file_metadata for the given sys_file was changed inside a workspace (that's why pid is set to -1).
With your update statement you generated a dead entry, that's why the workspace do not find it anymore.

So do we come back to your issue. You give us steps to reproduce the issue, does this reproduce the "sys_file_reference:XXX" issue or the "sys_file:XXX" issue?

#14 Updated by Rene Sonntag over 3 years ago

The steps describe a way to reproduce the "sys_file_reference:XXX" issue.

We can reproduce it with a clean TYPO3 installation and our project TYPO3 instances.

Thx, René

#15 Updated by Alexander Opitz over 3 years ago

  • Status changed from Needs Feedback to In Progress
  • Assignee set to Alexander Opitz

Fine, will check. Would be nice if you can find a way to reproduce the other one (or maybe it is a side effect of this one).

#16 Updated by Alexander Opitz over 3 years ago

  • Status changed from In Progress to Needs Feedback

Again a Question. :)

After the steps of reproducing and running the typo3/cli_dispatch.phpsh lowlevel_refindex -e command. Do you see something in the Workspace module for the "Live Workspace"?

#17 Updated by Rene Sonntag over 3 years ago

After the steps of reproducing the error "sys_file_reference:XXX" appears in Workspace module for the "Live Workspace".

After running typo3/cli_dispatch.phpsh lowlevel_refindex -e command we do see all the things that should be displayed - not an empty site or something like that.

René

#18 Updated by Alexander Opitz over 3 years ago

I mean, do you see elements in the Workspace module for the live Workspace? Normally you shouldn't see something there, as they are Live.

#19 Updated by Rene Sonntag over 3 years ago

In live workspace are two columns "Changed" and "Live-Title". Its filled with some information about former changes. Find a screenshot attached (clean TYPO3 6.2.13 with announced steps of reproducing).

In draft workspace there are "Changed", "Live-Title", "Current Stage" and "Actions".

René

#20 Updated by Alexander Opitz over 3 years ago

  • Status changed from Needs Feedback to In Progress

#21 Updated by Alexander Opitz over 3 years ago

At the moment I'm waiting for reviews on https://forge.typo3.org/issues/69135 ... which may also cause this issue.

#22 Updated by Alexander Opitz over 3 years ago

  • Status changed from In Progress to Needs Feedback
  • Assignee changed from Alexander Opitz to Rene Sonntag

Hi, the mentioned issue is now solved, can you please check your issue with TYPO3 from git?

#23 Updated by Rene Sonntag over 3 years ago

Hi,

I downloaded the latest version from git (TYPO3.CMS-cc04ac1, 75/43575/2, 6.2.16-dev), set up a clean instance and repeated the steps from the issue description above ... the issue still exists :-(

Best regards,
René

#24 Updated by Alexander Opitz over 3 years ago

  • Status changed from Needs Feedback to Accepted
  • Assignee changed from Rene Sonntag to Alexander Opitz

#25 Updated by Oliver Hader over 3 years ago

Ok, could reproduce with the mentioned steps in the workspace module in the live-workspace in 6.2.16-dev (TYPO3_6-2 branch)

#26 Updated by Oliver Hader over 3 years ago

Exception is thrown in ElementEntityProcessor::createNewDependentElementCallback
The missing sys_file_reference relations once was the delete-placeholder created in the edit action after the parent content element has been published the first time (according to the test-plan in the initial description of this issue). After fixing the reference index, the issue did no occur anymore.

This means:
  • reference index is not purged automatically on removing a delete-placeholder upon publishing
  • any far more important - the published workspace version is displayed in the workspace module of the live workspace, which should IMO not happen since it was a workspace version and is not a valid live version anymore (I'm referring to using the versioning feature without actually using workspaces)

#27 Updated by Oliver Hader over 3 years ago

  • Complexity set to medium

#28 Updated by Rene Sonntag over 3 years ago

As I mentioned above:

Only newer versions of TYPO3 has a problem in file handling in combination with workspaces. Last testet TYPO3 version without that issue is 6.2.6.

To fix the error 'Element "sys_file_reference:XXX" does not exist"' a "typo3/cli_dispatch.phpsh lowlevel_refindex -e" command will help.
To fix the error 'Element "sys_file:XXX" does not exist"' a "UPDATE sys_file_metadata SET pid='0' WHERE pid='-1';" SQL query will help.

Those workarounds ARE NOT applicable by customer. If somebody edit files on content elements in workspaces again - the error will recur. Issue should be fixed in TYPO3 Core.

None of our customer is willing to fix the reference index manually every time file references has been changed.

Its an issue of newer versions of TYPO3.

Thx, René

#29 Updated by Oliver Hader over 3 years ago

  • Target version set to next-patchlevel

#30 Updated by Mathias Schreiber over 3 years ago

  • Target version deleted (next-patchlevel)

#31 Updated by Rene Sonntag over 3 years ago

What are the next steps in this issue?

Best regards,
René

#32 Updated by Alexander Opitz over 3 years ago

  • Assignee changed from Alexander Opitz to Oliver Hader

#33 Updated by Daniel Schindler about 3 years ago

We ended up overwriting \TYPO3\CMS\Workspaces\ExtDirect\ActionHandler via XCLASS to cleanup the refindex table after relevant workspace actions.

ext_localconf.php:

$GLOBALS['TYPO3_CONF_VARS']['SYS']['Objects']['TYPO3\CMS\Workspaces\ExtDirect\ActionHandler'] = array(
    'className' => 'vendor\extensionname\XCLASS\ActionHandler'
);

typo3conf/ext/extensionname/Classes/XCLASS/ActionHandler.php

<?php
namespace vendor\extensionname\XCLASS;

use TYPO3\CMS\Backend\Utility\BackendUtility;
use TYPO3\CMS\Core\Utility\GeneralUtility;

/**
 * ...
 */
class ActionHandler extends \TYPO3\CMS\Workspaces\ExtDirect\ActionHandler
{

    /**
     * On all workspace actions, refresh the refindex table for affected datasets
     * ...
     */
    protected function processTcaCmd(array $cmdMapArray)
    {
        $result = parent::processTcaCmd($cmdMapArray);

        foreach ($cmdMapArray as $sTablename => $aRows) {
            if (is_array($aRows)) {
                foreach ($aRows as $iUidLive => $aVersionData) {
                    $iUidWorkspace = empty($aVersionData['version']['swapWith']) ? 0 : (int)$aVersionData['version']['swapWith'];
                    if ($iUidWorkspace) {
                        $this->updateRefIndexForWorkspaceAndLive($sTablename, $iUidLive);
                        $this->updateRefIndexForWorkspaceAndLive($sTablename, $iUidWorkspace);
                    }
                }
            }
        }

        return $result;
    }

    /**
     * If table is workspace activated, update Reference Index (sys_refindex) for a record in current and LIVE workspace
     * ...
     */
    protected function updateRefIndexForWorkspaceAndLive($table, $id)
    {
        /** @var $refIndexObj \TYPO3\CMS\Core\Database\ReferenceIndex */
        $refIndexObj = GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\Database\\ReferenceIndex');
        if (BackendUtility::isTableWorkspaceEnabled($table)) {
            $refIndexObj->setWorkspaceId(0);
            $refIndexObj->updateRefIndexTable($table, $id);

            if (!empty($GLOBALS['BE_USER']->workspace)) {
                $refIndexObj->setWorkspaceId($GLOBALS['BE_USER']->workspace);
                $refIndexObj->updateRefIndexTable($table, $id);
            }
        }
    }
}

#34 Updated by Martin Weymayer about 3 years ago

bug also comes up in type3 7.6.2. if i delete an image (logged in user without admin-rights) in draft workspace which was already aplied to "live workspace" an error occures:
router
Element "sys_file_reference:123" does not exist

if i go to workspaces live with admin rights. if i look up in table sys_file_reference uid 123 is missing.

#35 Updated by Andreas Krämer about 3 years ago

I have the same problem (TYPO3 6.2.12) and "typo3/cli_dispatch.phpsh lowlevel_refindex -e" could not solve it for 'Element "sys_file_reference:XXX" does not exist'. Even if I executed it multiple times.
So I deleted those entries manually to get workspaces to work again:

DELETE FROM sys_refindex WHERE ref_table = "sys_file_reference" AND ref_uid = XXX

#36 Updated by Andreas Lau about 3 years ago

Hey,

I tried to implement the XCLass-Solution (#33) but it did not work.
It either broke the whole typo3 with a message that it could not find the class "vendor\extensionname\XCLASS\ActionHandler" or the typo3 worked but there was no change in its behavior.
Do you have an idea if i missed something?

"typo3/cli_dispatch.phpsh lowlevel_refindex -e" works though.

Thank you for working on it.

#37 Updated by Christian Fries about 3 years ago

Same issue here with TYPO3 7.6.3

#38 Updated by Torsten Fink about 3 years ago

We have this problem all the time on the live servers, but with tt_content and pages (i think because of templavoila), so i get a nasty command to fix this in time until we found a solution

DELETE sysref.*
FROM sys_refindex sysref
WHERE hash IN (SELECT hash
               FROM (SELECT sr.hash
                     FROM `sys_refindex` sr LEFT JOIN tt_content c ON c.uid = sr.ref_uid WHERE sr.ref_table = 'tt_content' AND c.uid is null) x);

DELETE sysref.*
FROM sys_refindex sysref
WHERE hash IN (SELECT hash
               FROM (SELECT sr.hash
                     FROM `sys_refindex` sr LEFT JOIN pages p ON p.uid = sr.ref_uid WHERE sr.ref_table = 'pages' AND p.uid is null) x);

#39 Updated by Simon Praetorius about 3 years ago

I can reproduce this error message with TYPO3 6.2.19 doing the following:

  • Switch to Draft workspace
  • Upload Image1.jpg and Image2.jpg in file list
  • Create new page
  • Create new image element on page, select Image1.jpg, save
  • Edit image element, delete Image1.jpg reference, add Image2.jpg reference, save
  • Switch to file list, delete Image1.jpg (no error because it isn't referenced anymore)
  • Switch to Workspace module: "sys_file_reference:XXX" does not exist"

#40 Updated by Gerrit Code Review almost 3 years ago

  • Status changed from Accepted to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/47477

#41 Updated by Stefan Rotsch almost 3 years ago

IMHO this is a regression introduced with #64253.

Thanks to the detailed step description I was able to reproduce the issue with a functional test; please have a look at the patch set.

I'm in high doubt where to exactly put the test, but I hope you'll get the idea. Comments on how to clean this up highly appreciated.

The fix itself seems pretty straightforward, but I would love to hear if it does not interfere with the change intended by the author of #64253.

#42 Updated by Christian Weiske over 2 years ago

We see this in TYPO3 7.6.10

#43 Updated by Rafal Brzeski about 2 years ago

the same on 7.6.15

#44 Updated by Rene Sonntag almost 2 years ago

  • TYPO3 Version changed from 6.2 to 7

#45 Updated by Rene Sonntag almost 2 years ago

Same error on 7.6.16

#46 Updated by Riccardo De Contardi almost 2 years ago

Seems solved on 9.0.0-dev (latest master). (tested with test https://forge.typo3.org/issues/67676#note-39 )Can someone confirm?

#47 Updated by Frank Gerards over 1 year ago

The patch proposed triggers an error " [newlog()] Could not be edited in offline workspace in the branch where found (failure state: 'Workspace ID of record didn't match current workspace'). Auto-creation of version not allowed in workspace!" and the relation is completely lost during publishing into live-workspace (7.6.23).

Nvm - fixed as the initial relation state of these records was broken.
The patch proposed in this ticket seems to work for us on 7.6.23. Could you please integrate it into the core, so lots of users can profit from it ? I think the discussion about regression tests in this patch delaying it for 1-2 years is real unuseful.

#48 Updated by Gerrit Code Review over 1 year ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/47477

#49 Updated by Gerrit Code Review over 1 year ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/47477

#50 Updated by Sascha Schieferdecker over 1 year ago

I can confirm this bug still exists in 7.6.23. A fix would be very welcome.

#51 Updated by Benni Mack over 1 year ago

  • Sprint Focus set to On Location Sprint

#52 Updated by Gerrit Code Review about 1 year ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/47477

#53 Updated by Kurt Gusbeth about 1 year ago

The bug still exists in TYPO3 7.6.24.
Error message in the draft workspace:

router
Element "sys_file_reference:2408" does not exist

sys_log-entry for this case:

INSERT INTO `sys_log` (`uid`, `pid`, `userid`, `action`, `recuid`, `tablename`, `recpid`, `error`, `details`, `tstamp`, `type`, `details_nr`, `IP`, `log_data`, `event_pid`, `workspace`, `NEWid`, `request_id`, `time_micro`, `component`, `level`, `message`, `data`) VALUES
(22010, 0, 18, 1, 2408, 'sys_file_reference', 0, 0, 'New version created of table ''%s'', uid ''%s''. UID of new version is ''%s''', 1516182473, 1, 10, '195.145.62.234', 'a:3:{i:0;s:18:"sys_file_reference";i:1;i:934;i:2;i:2408;}', 331, 1, 'NEW5a5f1bc9bc091634568134', '', 0, '', 0, NULL, NULL),
(22014, 0, 18, 2, 2408, 'sys_file_reference', 0, 0, 'Record ''%s'' (%s) was updated. (Offline version).', 1516182473, 1, 10, '195.145.62.234', 'a:2:{i:0;s:17:"hope_seitlich.jpg";i:1;s:23:"sys_file_reference:2408";}', 331, 1, '', '', 0, '', 0, NULL, NULL),
(22016, 0, 18, 2, 2408, 'sys_file_reference', 0, 0, 'Record ''%s'' (%s) was updated. (Offline version).', 1516182583, 1, 10, '195.145.62.234', 'a:2:{i:0;s:17:"hope_seitlich.jpg";i:1;s:23:"sys_file_reference:2408";}', 331, 1, '', '', 0, '', 0, NULL, NULL),
(22018, 0, 18, 2, 2408, 'sys_file_reference', 0, 0, 'Record ''%s'' (%s) was updated. (Offline version).', 1516182646, 1, 10, '195.145.62.234', 'a:2:{i:0;s:17:"hope_seitlich.jpg";i:1;s:23:"sys_file_reference:2408";}', 331, 1, '', '', 0, '', 0, NULL, NULL),
(22071, 0, 18, 0, 2408, 'sys_file_reference', 0, 0, 'Stage for record was changed to -10. Comment was: "Hopi"', 1516189151, 1, -1, '195.145.62.234', 'a:0:{}', 331, 1, '', '', 0, '', 0, NULL, NULL),
(22072, 0, 18, 6, 2408, 'sys_file_reference', 0, 0, 'Stage raised...', 1516189151, 1, 30, '195.145.62.234', 'a:2:{s:7:"comment";s:4:"Hopi";s:5:"stage";i:-10;}', -1, 1, '', '', 0, '', 0, NULL, NULL),
(22088, 0, 15, 3, 2408, 'sys_file_reference', 0, 0, 'Record ''%s'' (%s) was deleted unrecoverable from page ''%s'' (%s)', 1516192603, 1, 0, '195.145.62.234', 'a:4:{i:0;s:12:"[Kein Titel]";i:1;s:23:"sys_file_reference:2408";i:2;s:12:"[root-level]";i:3;N;}', 0, 1, '', '', 0, '', 0, NULL, NULL),

#54 Updated by Kurt Gusbeth about 1 year ago

Note: we have sometimes the same problem with tt_content, but we have no templavoila in use.

#55 Updated by marco de felice about 1 year ago

the issue persist, but maybe exists only for LIVE work space, in other workspace the issue appear to fixed
typo3 7.6.24

#56 Updated by Mathias Brodala about 1 year ago

  • Sprint Focus deleted (On Location Sprint)

#57 Updated by Gerrit Code Review 5 months ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/47477

#58 Updated by Gerrit Code Review 3 months ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/47477

Also available in: Atom PDF