Bug #57103

FAL - Migrate file metadata from sys_file to an external metadata table fails with big data set

Added by Krystian Szymukowicz over 8 years ago. Updated almost 5 years ago.

Status:
Closed
Priority:
Should have
Assignee:
-
Category:
File Abstraction Layer (FAL)
Target version:
Start date:
2014-03-20
Due date:
% Done:

100%

Estimated time:
TYPO3 Version:
6.2
PHP Version:
5.4
Tags:
Complexity:
easy
Is Regression:
No
Sprint Focus:

Description

I have like 20k sys_files records.
While doing migrate file metadata from sys_file to sys_file_metadata in Install Tool I ended in "Fail".

It turns out that 20k files is too much to cope with.
Doing insertion by chunks was "Successful".

- class FileTableSplittingUpdate
- funkction checkForUpdate

....

$filesToMigrateUids = array_unique($filesToMigrateUids);
$filesToMigrateUidsParts = array_chunk($filesToMigrateUids, 100);
foreach($filesToMigrateUidsParts as $filesToMigrateUidsPart) {
          $dataToMove = $GLOBALS['TYPO3_DB']->exec_SELECTgetRows(
            implode(',', $this->detectFieldsToMigrate()) . ', uid AS file',
            'sys_file',
            'uid IN (' . implode(',', $filesToMigrateUidsPart) . ')'
            );
          $resultObject = $GLOBALS['TYPO3_DB']->exec_INSERTmultipleRows($this->metaDataTable, array_keys(current($dataToMove)), $dataToMove);
      if($resultObject === FALSE) {
        break;
    }
}
return $resultObject !== FALSE;
#1

Updated by Markus Klein over 8 years ago

Can you provide a patch?

#2

Updated by Gerrit Code Review over 8 years ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/28580

#3

Updated by Gerrit Code Review over 8 years ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/28580

#4

Updated by Gerrit Code Review over 8 years ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/28580

#5

Updated by Gerrit Code Review over 8 years ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/28580

#6

Updated by Gerrit Code Review over 8 years ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/28580

#7

Updated by Gerrit Code Review over 8 years ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/28580

#8

Updated by Krystian Szymukowicz over 8 years ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100
#9

Updated by Riccardo De Contardi almost 5 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF