Bug #62400

Lot of entries in sys_file_processed with name=NULL and identifier empty

Added by Albert van der Veen almost 7 years ago. Updated over 5 years ago.

Status:
Rejected
Priority:
Should have
Assignee:
Category:
File Abstraction Layer (FAL)
Target version:
Start date:
2014-10-22
Due date:
% Done:

0%

Estimated time:
TYPO3 Version:
6.2
PHP Version:
Tags:
Complexity:
easy
Is Regression:
No
Sprint Focus:

Description

We have a fairly large site with an enormous amount of entries in sys_file_processed that have name = NULL, identifier = ''. A substantional subset of those has configuration = a:9:{s:5:"width";N;s:6:"height";N;s:13:"fileExtension";N;s:8:"maxWidth";i:0;s:9:"maxHeight";i:0;s:8:"minWidth";i:0;s:9:"minHeight";i:0;s:7:"noScale";N;s:20:"additionalParameters";N;}

A related issue can be found here: https://forge.typo3.org/issues/50392#note-15

This leads to all kinds of problems, for example images not being displayed in the FE according to dimensions specified in a content element, but in the original size (no processed file is being generated). If the entry in sys_file_processed in manually removed and the FE page reloaded, the image is correctly displayed.

It also breaks the 'replace image' in Media (current 0.6.0-dev from the repo): as soon as a FE page is generated with an image a correct entry is added to sys_file_processed, but also an extra entry with name = NULL and configuration = a:9:{s:5:"width";N;s:6:"height";N;s:13:"fileExtension";N;s:8:"maxWidth";i:0;s:9:"maxHeight";i:0;s:8:"minWidth";i:0;s:9:"minHeight";i:0;s:7:"noScale";N;s:20:"additionalParameters";N;}

If you then use 'replace image' through Media on that image, the new image is not handled correctly: you don't get a new thumb and no output on FE. This can be traced back to the function flushProcessedFiles in media/Classes/Cache/CacheService.php which deletes old processed files. Because the entry with name=NULL exists, the newly uploaded (original, not processed) file is immediately deleted again. Just using 'replace image' again and again (without generating the FE page) works fine.

Changing $processedFile->delete(TRUE) to $processedFile->delete(FALSE) in that same function 'fixes' the problem, but I presume these NULL entries shouldn't be generated in the first place?

Anyone have an idea why these entries are generated?


Related issues

Related to TYPO3 Core - Bug #50392: Specifying size in ImageViewHelper does nothingClosed2013-07-24

Actions
Related to TYPO3 Core - Bug #61181: FAL: file maxW and maxH are ignoredRejected2014-08-25

Actions
Related to TYPO3 Core - Bug #59216: Image dimensions (width/height) are 0 when not scaledClosed2014-05-30

Actions
Related to TYPO3 Core - Bug #58019: FAL Indexer for broken files: Column 'width' cannot be nullRejected2014-04-17

Actions
Related to TYPO3 Core - Bug #54533: imgResource.noScale doesn't scale image tagRejected

Actions
Related to TYPO3 Core - Bug #62247: FAL: Uppercase File Extensions of Image cannot be displayed RejectedBenni Mack2014-10-15

Actions
Related to TYPO3 Core - Bug #44105: Image size does not get updatedClosed2012-12-19

Actions
Related to TYPO3 Core - Bug #45922: image replacement, width and height are kept even i change my imageClosed2013-02-28

Actions
Related to TYPO3 Core - Bug #46020: Image size is 0 when not scaledClosedAlexander Opitz2013-03-04

Actions
Related to TYPO3 Core - Bug #46446: sys_file doesn't get updatedClosed2013-03-20

Actions
Related to TYPO3 Core - Bug #52658: Overriding image does not change the dimensionsRejected2013-10-10

Actions
Related to FAL Indexing - Bug #60215: sys_file_processedfile with empty identifier and name fieldsUnder ReviewBenni Mack2014-07-09

Actions
Related to TYPO3 Core - Bug #64556: ProcessedFile record should not be saved/created when processing failedClosed2015-01-28

Actions
Related to TYPO3 Core - Bug #63519: sys_file_processedfile rows contain zero dimensionsClosed2014-12-02

Actions
Related to TYPO3 Core - Bug #89716: identifier not null in sys_file_processedfileClosed2019-11-20

Actions
#1

Updated by Albert van der Veen almost 7 years ago

Looking into the class TYPO3.CMS.Core.Resource.ProcessedFile, I now understand that entries with name=NULL are valid: 'A file may also meet the expectations set in the configuration without any processing. In that case, the ProcessedFile object still exists, but there is no physical file directly linked to it. Instead, it then redirects most method calls to the original file object. The data of these objects are also stored in the database, to indicate that no processing is required. With such files, the identifier and name fields in the database are empty to show this.'

But that still doesn't explain why these entries are generated for processed files. However, I suspect something is wrong with the config of this particular site, as I tested the same bahaviour on another site (far less complex as far as extensions etc is considered), and there no 'wrong' entries are generated. I'll investigate further and report back on my findings.

#2

Updated by Mathias Schreiber over 6 years ago

  • Target version set to 7.1 (Cleanup)
  • Sprint Focus set to On Location Sprint
#3

Updated by Frans Saris over 6 years ago

  • Complexity set to easy

possible solution

In typo3/sysext/core/Classes/Resource/Service/FileProcessingService::process() extend the check

 if ($processedFile->isProcessed()) {

That also checks if the identifier is set

#4

Updated by Benni Mack over 6 years ago

  • Target version changed from 7.1 (Cleanup) to 7.4 (Backend)
#5

Updated by Jonathan IROULIN over 6 years ago

  • Assignee set to Andreas Wolf

After discussion with Andreas Wolf, this is not a bug it is a feature.
This is the normal behaviour when the original file can be used,
to avoid to run the process file again and again.

#6

Updated by Andreas Wolf over 6 years ago

  • Status changed from New to Rejected
#7

Updated by Mathias Schreiber about 6 years ago

  • Sprint Focus deleted (On Location Sprint)
#8

Updated by Martin Kutschker over 5 years ago

If name is null and the identifier empty, what sould be the values of width and height? Shoudl they both be 0? I have seen this data and it broke the FE.

And there is also a fix concerning such entries:

https://review.typo3.org/#/c/39373/

#9

Updated by Andreas Wolf over 5 years ago

Martin Kutschker wrote:

If name is null and the identifier empty, what sould be the values of width and height? Shoudl they both be 0? I have seen this data and it broke the FE.

Of course the width and height of the original file should be used, as it can be used without processing (and the record is merely a flag to show that no processing is necessary and the processing attempt does not have to be repeated over and over again).

#10

Updated by Jonas Eberle over 5 years ago

We had this happen to a high extent (up to 20% of invalid records) on a very image-heavy site. It shows in the frontend as not rescaled images where the src-output of a <f:image>-Viewhelper has the original file instead of the processed file.

It can be seen in all Typo3-7.6 - instances that we have running so this is a bug with massive impact. Some instances use ImageMagick, some GraphicsMagick.

After 'clean up' -> 'clear processed files' in the InstallTool it would randomly happen again.

Here is a patch for typo3-7.6.4 (I am checking for the existence of 'identifier'):

diff -r typo3_src-7.6.4/typo3/sysext/core/Classes/Resource/ProcessedFileRepository.php typo3_src-7.6.4-processedFilePatched/typo3/sysext/core/Classes/Resource/ProcessedFileRepository.php
136,138c136,140
<             $this->databaseConnection->exec_INSERTquery($this->table, $insertFields);
<             $uid = $this->databaseConnection->sql_insert_id();
<             $processedFile->updateProperties(array('uid' => $uid));
---
>             if (isset($insertFields['identifier'])) {
>                 $this->databaseConnection->exec_INSERTquery($this->table, $insertFields);
>                 $uid = $this->databaseConnection->sql_insert_id();
>                 $processedFile->updateProperties(array('uid' => $uid));
>             }
154c156,158
<             $this->databaseConnection->exec_UPDATEquery($this->table, 'uid=' . (int)$uid, $updateFields);
---
>             if (isset($updateFields['identifier'])) {
>                  $this->databaseConnection->exec_UPDATEquery($this->table, 'uid=' . (int)$uid, $updateFields);
>             }

Sorry I did not check further. This is a workaround only. Finding the real culprit would be helpful of course.

#11

Updated by Florian Schuhmann almost 2 years ago

  • Related to Bug #89716: identifier not null in sys_file_processedfile added

Also available in: Atom PDF