Project

General

Profile

Actions
Copy link

Bug #80061

closed

FileStorageExtractionTask breaks if file not found

Added by Jonas Renggli about 7 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Should have
Assignee:
-
Category:
scheduler
Target version:
-
Start date:
2017-02-28
Due date:
% Done:

100%

Estimated time:
TYPO3 Version:
8
PHP Version:
Tags:
Complexity:
Is Regression:
No
Sprint Focus:

Description

Steps to Reproduce

  1. Ensure there's something to index. Add files, reset last_indexed field 
    UPDATE `sys_file`
SET `last_indexed` = 0;
  2. Delete file in filesystem
  3. Run scheduler Task "File Abstraction Layer: Extract metadata in storage" (FileStorageExtractionTask)
  4. Run scheduler Task "File Abstraction Layer: Update storage index" (FileStorageIndexingTask)
  5. Run scheduler Task "File Abstraction Layer: Extract metadata in storage" (FileStorageExtractionTask)

Actual Behavior

  • Error message 
    Task failed to execute successfully. Class: TYPO3\CMS\Scheduler\Task\FileStorageExtractionTask, UID: 4
  • Metadata for all other files won't be extracted

Expected Behavior

  • Skip missing file and continue with next file
  • Respect missing field in sys_file (set by FileStorageIndexingTask)

Files

Download all files
80061_typo3v8.patch (2.28 KB) 80061_typo3v8.patch Jonas Renggli, 2017-02-28 16:50
80061_typo3v7.patch (1.88 KB) 80061_typo3v7.patch Jonas Renggli, 2017-02-28 16:59
scheduler.png (26 KB) scheduler.png SICOR KDL GmbH, 2017-05-15 10:47

Related issues 1 (0 open — 1 closed)

Related to TYPO3 Core - Task #66217: [FAL] Show exception message or log exception when metadata extraction failClosedThomas Hohn2015-04-01

Actions
Actions
Copy link
 #1

Updated by Jonas Renggli about 7 years ago

  • Description updated (diff)

I see several ways to fix this issue:

  • Respect missing field in FileIndexRepository::findInStorageWithIndexOutstanding()
  • Catch Exception\FileDoesNotExistException and continue in Indexer::runMetaDataExtraction()
    • Optionally invoke FileIndexRepository::markFileAsMissing() in catch black
Actions
Copy link
 #2

Updated by Jonas Renggli about 7 years ago

Actions
Copy link
 #3

Updated by Jonas Renggli about 7 years ago

Actions
Copy link
 #4

Updated by Frans Saris about 7 years ago

Tnx for the patches. This looks indeed like the appropriate fix. Could you maybe push the patches to our review system? https://docs.typo3.org/typo3cms/ContributionWorkflowGuide/

Actions
Copy link
 #5

Updated by Jonas Renggli about 7 years ago

I'm currently testing the patches on a project with several thousand files. I'll push it to gerrit as soon as I'm sure it's working.
Unfortunately I don't have a project on TYPO3 v8 so far. Therefore I can't really test it. How is the workflow currently? Do I have to push the fix to master first and then backport it to v7?

Actions
Copy link
 #6

Updated by Wouter Wolters about 7 years ago

Have a look at this page https://docs.typo3.org/typo3cms/ContributionWorkflowGuide/
You should be able to get it in Gerrit with this documentation. If not, don't hesitate to contact us on Slack in channel #typo3-cms-coredev

To answer your question: master first :-)

Actions
Copy link
 #7

Updated by Thomas Hohn about 7 years ago

Hi Jonas

Tried to reproduce the error. The only case where I could trigger the error was when the record was
also missing in sys_file_metadata - could you maybe check if this is the case?

Actions
Copy link
 #8

Updated by Gerrit Code Review about 7 years ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969

Actions
Copy link
 #9

Updated by Gerrit Code Review about 7 years ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969

Actions
Copy link
 #10

Updated by SICOR KDL GmbH almost 7 years ago

Hello,

we experienced some problems in a typo3 8 upgrade with occasionally missing pictures and suspect it might come from the "Update storage index" task.

We experimented some with the info given here and the first thing we noticed is that sys_file entries with "storage=0" will never get indexed, they rest with "last_indexed = 0". Is that the expected behaviour?

Second thing we noticed is that even with parrallel execution set to "no", it looks from the logs as if it does nevertheless (I'll attach a screenshot). Can this break stuff when it runs in parallel with the "Extract metadata in storage" task?

Actions
Copy link
 #11

Updated by Gerrit Code Review almost 7 years ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969

Actions
Copy link
 #12

Updated by Gerrit Code Review over 6 years ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969

Actions
Copy link
 #13

Updated by Gerrit Code Review over 6 years ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969

Actions
Copy link
 #14

Updated by Gerrit Code Review over 6 years ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969

Actions
Copy link
 #15

Updated by Gerrit Code Review over 6 years ago

Patch set 1 for branch TYPO3_8-7 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/54647

Actions
Copy link
 #16

Updated by Thomas Hohn over 6 years ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100
Actions
Copy link
 #17

Updated by Benni Mack over 5 years ago

  • Status changed from Resolved to Closed
Actions
Copy link

Also available in: Atom PDF