Project

General

Profile

Actions

Bug #62829

closed

FAL not generating sys_file_metadata correctly

Added by Frederic Gaus about 10 years ago. Updated about 6 years ago.

Status:
Closed
Priority:
Should have
Assignee:
-
Category:
File Abstraction Layer (FAL)
Target version:
Start date:
2014-11-10
Due date:
% Done:

100%

Estimated time:
TYPO3 Version:
6.2
PHP Version:
Tags:
Complexity:
Is Regression:
No
Sprint Focus:

Description

Currently (tested on 6.2.6) FAL does not generate the metadata of images correctly. If an entry in sys_file_metadata is missing on file usage, FAL generates them on the fly. But an images' width/height are not correctly generated and thus are stored in database as "0".

This has a very silly behavior as a result. On the first uncached page view, the images are displayed correctly. This lasts until the second uncached page view is done. Since this moment, the images seem to be gone because they are generated with width=0 and height=0. After truncating sys_file_metadata the procedure can be reproduced.

This only occurs with images which are not processed by gifbuilder. To solve the problem there is a easy workaround:

UPDATE sys_file SET missing=1;

and then start the scheduler Task for indexing files (TYPO3\CMS\Scheduler\Task\FileStorageIndexingTask). But best would be to fix the reason.


Related issues 1 (0 open1 closed)

Related to TYPO3 Core - Bug #89020: Unknown column 'newlyCreated' in 'field list'Closed2019-08-26

Actions
Actions #1

Updated by Gerrit Code Review about 10 years ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/33990

Actions #2

Updated by Frans Saris about 10 years ago

  • Status changed from Under Review to Needs Feedback

Frederic Gaus wrote:

But best would be to fix the reason

Here I agree with you :)

Why are the sys_file_metadata records missing.
These are generated together with the sys_file record.

If you truncated the table yourself IMO running
UPDATE sys_file SET missing=1;

Wouldn't be a problem.

Actions #3

Updated by Frederic Gaus almost 10 years ago

I really can't tell why the the sys_file_metadata records are missing. The problem occurred in an updated TYPO3 version (6.1 -> 6.2). The problem only occurred in this instance. The records were not missing but the width and height was empty. The files where not added recently and there is no custom code at all involved in processing the files.

Of cause it would be easy to set "UPDATE sys_file SET missing=1 - but this is exactly the problem. If you do and start the indexer the meta data is regenerated with width and height = 0 and not with the correct values. So this does not help in this situation.

Actions #4

Updated by Sybille Peters almost 10 years ago

We also had this problem with width and height=0 in sys_file_metadata. This was on an installation which started with an early version of 6.0 or 6.1 and had been updated. Files had been added continuously on the server and the FAL indexer run regularly.

Also, there were duplicate entries (2 entries with the same identifier and the same storage id in sys_file).

Actions #5

Updated by Mathias Brodala almost 10 years ago

One reason sys_file_metadata records could be missing is an upgrade from 6.1 where there only was sys_file.

Actions #6

Updated by Frederic Gaus almost 10 years ago

Our instance was also an updated one (6.1 to 6.2)

Actions #7

Updated by Frans Saris almost 10 years ago

Did you run all the upgrade wizards correct when upgrading to 6.2?

Of cause it would be easy to set "UPDATE sys_file SET missing=1 - but this is exactly the problem. If you do and start the indexer the meta data is regenerated with width and height = 0 and not with the correct values. So this does not help in this situation.

Could it be that the width and height columns of the sys_file table are still present? These need to be removed. The width and height columns are moved to sys_file_metadata since 6.2.

Actions #8

Updated by Sergio Catalá almost 10 years ago

My instance was updated from 6.1 to 6.2 too.
Same problem here.
All workarounds in this issue and #46020 were applied without success.
I built a script based on #46020#note-81 that executes sql updates over sys_file_metadata using "identify" command, but not all images were covered.

Any solution for this?

Actions #9

Updated by Frans Saris almost 10 years ago

You build a script that looks in all folders that are indexed?

Do you see a common difference between the sys_file_metadata records that are correct now and the once that aren't? Same folder or storage uid?

Does your sys_file table still have the width and height column? Because that would result in some issues.

Actions #10

Updated by Sergio Catalá almost 10 years ago

Frans Saris wrote:

You build a script that looks in all folders that are indexed?

Correct. I have three "storages": uploads/, fileadmin/ and "the rest". And my scripts goes for all pictures (png,jpg,gif), identifies the width and the height for every image and builds a sql update for every image in the sys_file_metadata table.

Do you see a common difference between the sys_file_metadata records that are correct now and the once that aren't? Same folder or storage uid?

No. No pattern as far as I can see.

Does your sys_file table still have the width and height column? Because that would result in some issues.

No. After applying the upgrade wizard in the backend, I applied the compare tool and deleted these fields.
I would like to recreate them again but in the sys_file_metadata table.
The scheduler tasks referring to FAL didn't work for me.
Width and height are setup to 0 again.

Actions #11

Updated by Frans Saris almost 10 years ago

Some more questions then:
  • what filesystem are you using? *nix, mac, windows
  • Are your filestorages set to case-sensitive
  • You got a storage 0, 1 and 2? 0 = the rest, 1 = uploads, 2 = fileadmin?
  • If you fetch a file that doesn't get a width and height, rename it and upload it in BE filelist the width and height are set correct?
  • If your script identifies the width and height of every png, jpg and gif image. Why are the width and height still 0 than? Is the file identifier somehow incorrect?

gr. Frans

Actions #12

Updated by Sergio Catalá almost 10 years ago

Frans Saris wrote:

  • what filesystem are you using? *nix, mac, windows

Unix.

  • Are your filestorages set to case-sensitive

Yes.

  • You got a storage 0, 1 and 2? 0 = the rest, 1 = uploads, 2 = fileadmin?

Yes.

  • If you fetch a file that doesn't get a width and height, rename it and upload it in BE filelist the width and height are set correct?

Yes.

  • If your script identifies the width and height of every png, jpg and gif image. Why are the width and height still 0 than? Is the file identifier somehow incorrect?

The file identifier is correct. Width and height are setup properly with my script but I can't get all images to have proper width and height, as "identify" command fails for some that are corrupted.

What I am worried is that, even when all are setup properly, refreshing the cache makes them back to 0 width and height again.

Actions #13

Updated by Frans Saris almost 10 years ago

Refreshing the cache will not touch the sys_file_metadata record.

The "required" metadata will only be re-set when the sys_file record is reindex because of the missing flag or filesize change.

Could you maybe do some further debugging and have a look at the result of list($metaData['width'], $metaData['height']) = getimagesize($rawFileLocation); in TYPO3\CMS\Core\Resource\Index\Indexer::extractRequiredMetaData().

Flag a sys_file missing and run the FAL indexer task.

Actions #14

Updated by Sergio Catalá almost 10 years ago

As you recommended, I made some debugging and I could see that the scheduler task fills properly the dimensions. That allowed me to observe something: images outside of folders "fileadmin" and "uploads" are never indexed and filled with the proper dimensions.

I created two scheduler tasks for every storage in my system ("fileadmin" and "uploads") and ran them.
Everything went ok.
But what about images that are used in my extensions under typo3conf/ ?
My workaround was creating a new "storage" on PID=0 and create a new scheduler task for indexing this storage.
Now the image is twice in sys_file_metadata table, but one row is for storage 0 (all), which has width=height=0, and the other is for my new storage, that sets up the proper dimensions. The frontend takes this one.

I confirm the original bug commented still exists:

1. If sys_file_metadata table is truncated for some reason (or the upgrade tool doesn't work well), the first time the frontend is refreshed, the images are shown. They don't have width and height parameters at all.

2. New sys_file_metadata rows for these images are created with width=height=0.

3. The user refreshes the cache and then the frontend displays no images as they take the information from sys_file_metadata.

Running this new scheduler task with a "storage" for typo3conf/ solves the problem for me.

Actions #15

Updated by Frans Saris almost 10 years ago

  • Status changed from Needs Feedback to New
  • Sprint Focus set to On Location Sprint

I would not recommend to create a storage for typo3conf. As a temporary hotfix for the storage 0 problem you could just delete all records in sys_file with storage = 0 as these files will only be used by their identifier and not by their uid.

For re-indexing storage 0 files there is already an issue #64196.

Hope to resolve #64196 and this issue "New sys_file_metadata rows for these images are created with width=height=0." with the FAL codesprint next week.

Actions #16

Updated by Frans Saris almost 10 years ago

  • Status changed from New to Under Review
Actions #17

Updated by Sergio Catalá almost 10 years ago

Thank you very much for the feedback, Frans.

Actions #18

Updated by Gerrit Code Review almost 10 years ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/33990

Actions #19

Updated by Gerrit Code Review almost 10 years ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/33990

Actions #20

Updated by Gerrit Code Review almost 10 years ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/33990

Actions #21

Updated by Gerrit Code Review almost 10 years ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/33990

Actions #22

Updated by Gerrit Code Review almost 10 years ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/33990

Actions #23

Updated by Gerrit Code Review almost 10 years ago

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/33990

Actions #24

Updated by Gerrit Code Review almost 10 years ago

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/36454

Actions #25

Updated by Gerrit Code Review almost 10 years ago

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/36457

Actions #26

Updated by Frederic Gaus almost 10 years ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100
Actions #27

Updated by Gerrit Code Review almost 10 years ago

  • Status changed from Resolved to Under Review

Patch set 2 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/36457

Actions #28

Updated by Anja Leichsenring almost 10 years ago

  • Status changed from Under Review to Resolved
Actions #29

Updated by Sven Burkert almost 10 years ago

Thank you, the hint with "UPDATE sys_file SET missing=1" helped.

This also worked for me:

UPDATE sys_file_metadata fm, sys_file f SET fm.width = f.width WHERE fm.file = f.uid;
UPDATE sys_file_metadata fm, sys_file f SET fm.height = f.height WHERE fm.file = f.uid;

Actions #30

Updated by Frans Saris almost 10 years ago

The width and height columns of sys_file should be removed (see install tool => DB compare). Else you keep running into this issue.

Actions #31

Updated by Anja Leichsenring almost 9 years ago

  • Sprint Focus deleted (On Location Sprint)
Actions #32

Updated by Benni Mack about 6 years ago

  • Status changed from Resolved to Closed
Actions #33

Updated by Oliver Hader almost 5 years ago

  • Related to Bug #89020: Unknown column 'newlyCreated' in 'field list' added
Actions

Also available in: Atom PDF