Bug #82060

File Abstraction Layer: Extract metadata in storage (scheduler) failing on indexing files that matches denied patterns(['BE']['fileDenyPattern'])

Added by Ricky Mathew 4 months ago. Updated 25 days ago.

Status:
Resolved
Priority:
Must have
Assignee:
Category:
scheduler
Start date:
2017-08-09
Due date:
% Done:

100%

TYPO3 Version:
7
PHP Version:
7.1
Tags:
Complexity:
easy
Is Regression:
Sprint Focus:

Description

File Abstraction Layer: Extract metadata in storage (scheduler) fails on trying to index the files in denied pattern(['BE']['fileDenyPattern']) by throwing an exception like following::

_TYPO3\CMS\Core\Resource\Exception\InsufficientFileAccessPermissionsExceptionprototypeobject
message => 'You are not allowed to access that file: "browserdetect.inc.php"' (64 chars)
code => 1375955429 (integer)
_

And my ['BE']['fileDenyPattern'] is "\.(php[3-7]?|phpsh|phtml)(\..*)?$|^\.htaccess$"

We can avoid this by just adding a condition check in runMetaDataExtraction() function in TYPO3\CMS\Core\Resource\Index\Indexer class as follows

$isAllowed = GeneralUtility::verifyFilenameAgainstDenyPattern($indexRecord['name']);

           if($isAllowed){
            $fileObject = $this->getResourceFactory()->getFileObject($indexRecord['uid'], $indexRecord);
            $this->extractMetaData($fileObject);
           }

so can any one please provide a patch with above mentioned solution?

Indexer_patch.patch View (695 Bytes) Ricky Mathew, 2017-08-10 07:31

Associated revisions

Revision 65a05362 (diff)
Added by Ricky Mathew 25 days ago

[BUGFIX] Catch Exception while extracting metadata

Catch InsufficientFileAccessPermissionsException while extracting
metadata. Reason for this exception could be that the file extension is
not allowed by the ['BE']['fileDenyPattern'] setting.

This patch makes sure that the indexer doesn't break and is able to
extract the other files from storage.

Resolves: #82060
Release: master, 8.7, 7.6
Change-Id: I37950aa70f4f59b388c6b9203e1922708ad07b71
Reviewed-on: https://review.typo3.org/53834
Tested-by: TYPO3com <>
Reviewed-by: Ricky Mathew <>
Tested-by: Ricky Mathew <>
Reviewed-by: Alexander Opitz <>
Reviewed-by: Wouter Wolters <>
Reviewed-by: Georg Ringer <>
Tested-by: Georg Ringer <>

Revision 4228c7c5 (diff)
Added by Ricky Mathew 18 days ago

[BUGFIX] Catch Exception while extracting metadata

Catch InsufficientFileAccessPermissionsException while extracting
metadata. Reason for this exception could be that the file extension is
not allowed by the ['BE']['fileDenyPattern'] setting.

This patch makes sure that the indexer doesn't break and is able to
extract the other files from storage.

Resolves: #82060
Release: master, 8.7, 7.6
Change-Id: I37950aa70f4f59b388c6b9203e1922708ad07b71
Reviewed-on: https://review.typo3.org/54708
Tested-by: TYPO3com <>
Reviewed-by: Wouter Wolters <>
Tested-by: Wouter Wolters <>

History

#1 Updated by Ricky Mathew 4 months ago

  • Complexity changed from hard to easy

#2 Updated by Ricky Mathew 4 months ago

The logic is , we need to run the extractMetaData() function only if the file is of allowable type and that is what the condition check will do.

#3 Updated by Ricky Mathew 4 months ago

  • Category set to scheduler
  • Target version set to Candidate for patchlevel

#4 Updated by Ricky Mathew 4 months ago

Attaching the patch file here.

#5 Updated by Ricky Mathew 4 months ago

--- Indexer_orig.php    2017-08-09 16:02:52.912607837 +0530
+++ Indexer.php    2017-08-09 16:10:08.404625955 +0530
@@ -106,8 +106,12 @@
     {
         $fileIndexRecords = $this->getFileIndexRepository()->findInStorageWithIndexOutstanding($this->storage, $maximumFileCount);
         foreach ($fileIndexRecords as $indexRecord) {
+           $isAllowed = GeneralUtility::verifyFilenameAgainstDenyPattern($indexRecord['name']);
+           if ($isAllowed) {
             $fileObject = $this->getResourceFactory()->getFileObject($indexRecord['uid'], $indexRecord);
             $this->extractMetaData($fileObject);
+           }
         }
     }

#6 Updated by Ricky Mathew 4 months ago

  • Assignee set to Ricky Mathew

#7 Updated by Ricky Mathew 4 months ago

As my membership in core project is still pending, can anyone please take this task and apply this patch?

#8 Updated by Frans Saris 4 months ago

Hi Ricky,

tnx for the patch. You don't have to be a member of the core to push patches to Gerrit. Like most other OS projects everyone is allowed to push patches.

https://docs.typo3.org/typo3cms/ContributionWorkflowGuide/

Please let us know if you need further assistance with pushing this to Gerrit. On our Slack channel there is always someone around that's willing to assist you. https://forger.typo3.org/slack

#9 Updated by Frans Saris 4 months ago

btw maybe is't easier to just catch the InsufficientFileAccessPermissionsException. So add a

try { 
 ...
} catch(InsufficientFileAccessPermissionsException $e) {}

arround the call

#10 Updated by Ricky Mathew 4 months ago

Frans Saris wrote:

btw maybe is't easier to just catch the InsufficientFileAccessPermissionsException. So add a

[...]

arround the call

But that will throw the exception and stops the scheduler execution..But we just need to skip the denied files. that's all

#11 Updated by Frans Saris 4 months ago

No it does mainly the same as your solution

         $fileIndexRecords = $this->getFileIndexRepository()->findInStorageWithIndexOutstanding($this->storage, $maximumFileCount);
         foreach ($fileIndexRecords as $indexRecord) {
           try {
             $fileObject = $this->getResourceFactory()->getFileObject($indexRecord['uid'], $indexRecord);
             $this->extractMetaData($fileObject);
           } catch(InsufficientFileAccessPermissionsException $e) {}
         }

It will still loop thought all items

#12 Updated by Ricky Mathew 4 months ago

Frans Saris wrote:

No it does mainly the same as your solution

[...]

It will still loop thought all items

Okay.Let me check whether it works for me.

#13 Updated by Gerrit Code Review about 1 month ago

  • Status changed from New to Under Review

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53834

#14 Updated by Gerrit Code Review 27 days ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53834

#15 Updated by Gerrit Code Review 26 days ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/53834

#16 Updated by Gerrit Code Review 25 days ago

Patch set 1 for branch TYPO3_8-7 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/54708

#17 Updated by Ricky Mathew 25 days ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

Also available in: Atom PDF