Bug #78845

Story #69712: Further FormEngine development

Editing a sys_file_metadata record (eg. through filelist) fetches all sys_file records

Added by Sascha Nowak almost 4 years ago. Updated about 2 years ago.

Status:
Closed
Priority:
Must have
Assignee:
-
Category:
File Abstraction Layer (FAL)
Target version:
-
Start date:
2016-11-30
Due date:
% Done:

100%

Estimated time:
TYPO3 Version:
7
PHP Version:
7.0
Tags:
Complexity:
Is Regression:
No
Sprint Focus:

Description

After some discussion on Slack with lolli, he wrote the following:

Analysis:
  • field "file" in "sys_file_metadata" is configured as select field to sys_file. it is used as: the "inline" counterpart field of field "sys_file" "metadata" since the fields holds the uid of the sys_file record, this sys_file_metadata is for.
  • during opening a sys_file_metadata record (eg. through filelist), the TcaSelectItems data provider fetches all "possible" select targets (which basically means "all" sys_file records) with one single query. this can become "a lot" - 10MB for 10k records
  • the fun part is, the file field itself is never displayed (not rendered), since it is a) set to readonly, b) shut down per display condition for sys_language_uid!=0, c) it is in NO showitem configuration
  • however, it is important to submit the current field value to the data handler, since otherwise the relation to sys_file would get lost upon editing a sys_file_metadata record
Possible solutions:
  • it "may" indeed be possible to switch the field to a group field to suppress fetching these records all the time
Questions / Scenarios:
  • we need to verify that the inline "from sys_file to sys_file_metadata" direction still works if the counter side is switched from select to group
  • we still should mark the field "readOnly" for group, is that possible?
  • does the "placeholder" / "override" stuff still work if switched group, for instance in tt_content->sys_file->sys_file_metadata ? it is very well possible warnings are raised here since the internal select and group data handling is slightly different (select=array, group=comma mess) ... i remember the placeholder logic is also funny and recursive and this needs an additional check then.
  • currently, if doing tt_content editings ... i "think" the calculation of that "sys_file_metadata" "file" field item calculation is probably suppressed already somehow (we had massive performance issues in fal with new FormEngine, and I somehow solved that by suppressing some calculation - question is "how"?). this should have a detailed look again.
  • does stuff like the fal indexer and uploader work successfully with group and fills / updates metadata correctly and sets correct relations? (edited)
Another possible solution I could think about is
  • Ignore fetching records in the TcaSelectItems for readOnly fields (which may not work if the field is used in relations)

Related issues

Has duplicate TYPO3 Core - Bug #83308: Allowed memory size exhausted when editing file meta dataClosed2017-12-13

Actions
#1

Updated by Christian Kuhn almost 4 years ago

  • Description updated (diff)
#2

Updated by Christian Kuhn almost 4 years ago

  • Parent task set to #69712
#3

Updated by Markus Klein almost 4 years ago

This is the same issue we had with normal records and translations. The l10n_parent by default is a select field as well. Very naughty if you have 25k records on the same page, which all might be the a potential language parent.
We changed TCA to group here to get rid of the issue.

#4

Updated by Wladimir Hofmann over 3 years ago

The column "file" in TCA"sys_file_metadata" has a display condition:

'displayCond' => 'FIELD:sys_language_uid:=:0'

Doesn't it make sense to add an additional

'foreign_table_where' =>  'AND sys_file.uid=###REC_FIELD_file###'

condition as well?

worked for me, without side effects so far.

#5

Updated by DANIEL Rémy over 3 years ago

Hello

Adding the foreign_table_where clause worked for me too.

The query array before:

   SELECT => 'sys_file.uid,sys_file.name,sys_file.type'
   FROM => 'sys_file'
   WHERE => 'sys_file.pid<>-1 '
   GROUPBY => ''
   ORDERBY => ''
   LIMIT => ''

The query array after:

   SELECT => 'sys_file.uid,sys_file.name,sys_file.type'
   FROM => 'sys_file'
   WHERE => 'sys_file.pid<>-1 AND sys_file.uid='11499''
   GROUPBY => ''
   ORDERBY => ''
   LIMIT => ''

I cannot see either any side effects with this fix.

#6

Updated by Torben Hansen over 3 years ago

In my case, the FAL contains 300.000+ records. Memory usage of the edit file metadata request raised to 500 MB and rendering took 10-30 seconds depending on the webserver.

The foreign_table_where seems to be a good solution for this scenario.

#7

Updated by Gerrit Code Review almost 3 years ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/55125

#8

Updated by Gerrit Code Review almost 3 years ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/55125

#9

Updated by Stephan Großberndt almost 3 years ago

  • Has duplicate Bug #83308: Allowed memory size exhausted when editing file meta data added
#10

Updated by Gerrit Code Review almost 3 years ago

Patch set 1 for branch TYPO3_8-7 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/55167

#11

Updated by Gerrit Code Review almost 3 years ago

Patch set 1 for branch TYPO3_7-6 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/55168

#12

Updated by Stephan Großberndt almost 3 years ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100
#13

Updated by Benni Mack about 2 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF