Bug #80061

FileStorageExtractionTask breaks if file not found

Added by Jonas Renggli over 2 years ago. Updated 12 months ago.

Status:
Closed
Priority:
Should have
Assignee:
-
Category:
scheduler
Target version:
-
Start date:
2017-02-28
Due date:
% Done:

100%

TYPO3 Version:
8
PHP Version:
Tags:
Complexity:
Is Regression:
No
Sprint Focus:

Description

Steps to Reproduce

  1. Ensure there's something to index. Add files, reset last_indexed field
    UPDATE `sys_file`
    SET `last_indexed` = 0;
    
  2. Delete file in filesystem
  3. Run scheduler Task "File Abstraction Layer: Extract metadata in storage" (FileStorageExtractionTask)
  4. Run scheduler Task "File Abstraction Layer: Update storage index" (FileStorageIndexingTask)
  5. Run scheduler Task "File Abstraction Layer: Extract metadata in storage" (FileStorageExtractionTask)

Actual Behavior

  • Error message
    Task failed to execute successfully. Class: TYPO3\CMS\Scheduler\Task\FileStorageExtractionTask, UID: 4
    
  • Metadata for all other files won't be extracted

Expected Behavior

  • Skip missing file and continue with next file
  • Respect missing field in sys_file (set by FileStorageIndexingTask)

80061_typo3v8.patch View (2.28 KB) Jonas Renggli, 2017-02-28 16:50

80061_typo3v7.patch View (1.88 KB) Jonas Renggli, 2017-02-28 16:59

scheduler.png View (26 KB) SICOR KDL GmbH, 2017-05-15 10:47


Related issues

Related to TYPO3 Core - Task #66217: [FAL] Show exception message or log exception when metadata extraction fail Closed 2015-04-01

Associated revisions

Revision 9ba9c878 (diff)
Added by Thomas Hohn almost 2 years ago

[BUGFIX] FileStorageExtractionTask breaks if file not found

Before the file is processed a extra check is done to be sure the file
exists. If the file doesn't exist it's marked as missing.

Resolves: #80061
Releases: master, 8.7
Change-Id: Ifda396b75d2ace0ce126057a4e118856b15c05e0
Reviewed-on: https://review.typo3.org/51969
Reviewed-by: Frans Saris <>
Tested-by: Frans Saris <>
Tested-by: TYPO3com <>
Reviewed-by: Thomas Hohn <>
Reviewed-by: Markus Klein <>
Tested-by: Markus Klein <>

Revision dd4cf172 (diff)
Added by Thomas Hohn almost 2 years ago

[BUGFIX] FileStorageExtractionTask breaks if file not found

Before the file is processed a extra check is done to be sure the file
exists. If the file doesn't exist it's marked as missing.

Resolves: #80061
Releases: master, 8.7
Change-Id: Ifda396b75d2ace0ce126057a4e118856b15c05e0
Reviewed-on: https://review.typo3.org/54647
Tested-by: TYPO3com <>
Reviewed-by: Markus Klein <>
Tested-by: Markus Klein <>

History

#1 Updated by Jonas Renggli over 2 years ago

  • Description updated (diff)

I see several ways to fix this issue:

  • Respect missing field in FileIndexRepository::findInStorageWithIndexOutstanding()
  • Catch Exception\FileDoesNotExistException and continue in Indexer::runMetaDataExtraction()
    • Optionally invoke FileIndexRepository::markFileAsMissing() in catch black

#2 Updated by Jonas Renggli over 2 years ago

#3 Updated by Jonas Renggli over 2 years ago

#4 Updated by Frans Saris over 2 years ago

Tnx for the patches. This looks indeed like the appropriate fix. Could you maybe push the patches to our review system? https://docs.typo3.org/typo3cms/ContributionWorkflowGuide/

#5 Updated by Jonas Renggli over 2 years ago

I'm currently testing the patches on a project with several thousand files. I'll push it to gerrit as soon as I'm sure it's working.
Unfortunately I don't have a project on TYPO3 v8 so far. Therefore I can't really test it. How is the workflow currently? Do I have to push the fix to master first and then backport it to v7?

#6 Updated by Wouter Wolters over 2 years ago

Have a look at this page https://docs.typo3.org/typo3cms/ContributionWorkflowGuide/
You should be able to get it in Gerrit with this documentation. If not, don't hesitate to contact us on Slack in channel #typo3-cms-coredev

To answer your question: master first :-)

#7 Updated by Thomas Hohn over 2 years ago

Hi Jonas

Tried to reproduce the error. The only case where I could trigger the error was when the record was
also missing in sys_file_metadata - could you maybe check if this is the case?

#8 Updated by Gerrit Code Review over 2 years ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969

#9 Updated by Gerrit Code Review over 2 years ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969

#10 Updated by SICOR KDL GmbH over 2 years ago

Hello,

we experienced some problems in a typo3 8 upgrade with occasionally missing pictures and suspect it might come from the "Update storage index" task.

We experimented some with the info given here and the first thing we noticed is that sys_file entries with "storage=0" will never get indexed, they rest with "last_indexed = 0". Is that the expected behaviour?

Second thing we noticed is that even with parrallel execution set to "no", it looks from the logs as if it does nevertheless (I'll attach a screenshot). Can this break stuff when it runs in parallel with the "Extract metadata in storage" task?

#11 Updated by Gerrit Code Review about 2 years ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969

#12 Updated by Gerrit Code Review almost 2 years ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969

#13 Updated by Gerrit Code Review almost 2 years ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969

#14 Updated by Gerrit Code Review almost 2 years ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969

#15 Updated by Gerrit Code Review almost 2 years ago

Patch set 1 for branch TYPO3_8-7 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/54647

#16 Updated by Thomas Hohn almost 2 years ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

#17 Updated by Benni Mack 12 months ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF