Bug #80061
closedFileStorageExtractionTask breaks if file not found
100%
Description
Steps to Reproduce¶
- Ensure there's something to index. Add files, reset
last_indexed
fieldUPDATE `sys_file` SET `last_indexed` = 0;
- Delete file in filesystem
- Run scheduler Task "File Abstraction Layer: Extract metadata in storage" (
FileStorageExtractionTask
) - Run scheduler Task "File Abstraction Layer: Update storage index" (
FileStorageIndexingTask
) - Run scheduler Task "File Abstraction Layer: Extract metadata in storage" (
FileStorageExtractionTask
)
Actual Behavior¶
- Error message
Task failed to execute successfully. Class: TYPO3\CMS\Scheduler\Task\FileStorageExtractionTask, UID: 4
- Metadata for all other files won't be extracted
Expected Behavior¶
- Skip missing file and continue with next file
- Respect
missing
field insys_file
(set byFileStorageIndexingTask
)
Files
Updated by Jonas Renggli over 7 years ago
- Description updated (diff)
I see several ways to fix this issue:
- Respect
missing
field inFileIndexRepository::findInStorageWithIndexOutstanding()
- Catch
Exception\FileDoesNotExistException
and continue inIndexer::runMetaDataExtraction()
- Optionally invoke
FileIndexRepository::markFileAsMissing()
in catch black
- Optionally invoke
Updated by Jonas Renggli over 7 years ago
- File 80061_typo3v8.patch 80061_typo3v8.patch added
- Description updated (diff)
Updated by Jonas Renggli over 7 years ago
- File 80061_typo3v7.patch 80061_typo3v7.patch added
- Description updated (diff)
Updated by Frans Saris over 7 years ago
Tnx for the patches. This looks indeed like the appropriate fix. Could you maybe push the patches to our review system? https://docs.typo3.org/typo3cms/ContributionWorkflowGuide/
Updated by Jonas Renggli over 7 years ago
I'm currently testing the patches on a project with several thousand files. I'll push it to gerrit as soon as I'm sure it's working.
Unfortunately I don't have a project on TYPO3 v8 so far. Therefore I can't really test it. How is the workflow currently? Do I have to push the fix to master first and then backport it to v7?
Updated by Wouter Wolters over 7 years ago
Have a look at this page https://docs.typo3.org/typo3cms/ContributionWorkflowGuide/
You should be able to get it in Gerrit with this documentation. If not, don't hesitate to contact us on Slack in channel #typo3-cms-coredev
To answer your question: master first :-)
Updated by Thomas Hohn over 7 years ago
Hi Jonas
Tried to reproduce the error. The only case where I could trigger the error was when the record was
also missing in sys_file_metadata - could you maybe check if this is the case?
Updated by Gerrit Code Review over 7 years ago
- Status changed from New to Under Review
Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969
Updated by Gerrit Code Review over 7 years ago
Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969
Updated by SICOR KDL GmbH over 7 years ago
- File scheduler.png scheduler.png added
Hello,
we experienced some problems in a typo3 8 upgrade with occasionally missing pictures and suspect it might come from the "Update storage index" task.
We experimented some with the info given here and the first thing we noticed is that sys_file entries with "storage=0" will never get indexed, they rest with "last_indexed = 0". Is that the expected behaviour?
Second thing we noticed is that even with parrallel execution set to "no", it looks from the logs as if it does nevertheless (I'll attach a screenshot). Can this break stuff when it runs in parallel with the "Extract metadata in storage" task?
Updated by Gerrit Code Review over 7 years ago
Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969
Updated by Gerrit Code Review almost 7 years ago
Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969
Updated by Gerrit Code Review almost 7 years ago
Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969
Updated by Gerrit Code Review almost 7 years ago
Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/51969
Updated by Gerrit Code Review almost 7 years ago
Patch set 1 for branch TYPO3_8-7 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/54647
Updated by Thomas Hohn almost 7 years ago
- Status changed from Under Review to Resolved
- % Done changed from 0 to 100
Applied in changeset 9ba9c8780af50c239338419f0d6568cd56d006d1.