Project

General

Profile

Actions

Bug #63668

closed

Bug #63692: Memory consumption while bulk inserting

High memory consumption in DataHandler->fillInFieldArray while bulk inserting

Added by Stephan Großberndt almost 10 years ago. Updated about 6 years ago.

Status:
Closed
Priority:
Must have
Assignee:
-
Category:
Performance
Target version:
Start date:
2014-12-08
Due date:
% Done:

100%

Estimated time:
TYPO3 Version:
6.2
PHP Version:
Tags:
Complexity:
easy
Is Regression:
No
Sprint Focus:

Description

DataHandler->fillInFieldArray calls

$eFile = \TYPO3\CMS\Core\Html\RteHtmlParser::evalWriteFile($vconf['spec']['static_write'], array_merge($currentRecord, $fieldArray));

for each field of each record to insert. In each call array_merge($currentRecord, $fieldArray) is done only to throw it away again immediately and do nothing in most cases - does anyone use staticFileEditPath? I dont't think so: http://forum.typo3.org/index.php/t/195196/

$types_fieldConfig = BackendUtility::getTCAtypes($table, $currentRecord);
$theTypeString = BackendUtility::getTCAtypeValue($table, $currentRecord);
if (is_array($types_fieldConfig)) {
    foreach ($types_fieldConfig as $vconf) {
        // Write file configuration:
        // inserted array_merge($currentRecord,$fieldArray) 170502
        $eFile = \TYPO3\CMS\Core\Html\RteHtmlParser::evalWriteFile($vconf['spec']['static_write'], array_merge($currentRecord, $fieldArray));
        // RTE transformations:
        if (!$this->dontProcessTransformations) {
}

Instead of unconditionally merging and throwing away again only merge if static_write is enabled

$types_fieldConfig = BackendUtility::getTCAtypes($table, $currentRecord);
$theTypeString = BackendUtility::getTCAtypeValue($table, $currentRecord);
if (is_array($types_fieldConfig)) {
    foreach ($types_fieldConfig as $vconf) {
        if (is_array($vconf['spec']['static_write']) && $GLOBALS['TYPO3_CONF_VARS']['BE']['staticFileEditPath']) {
            // only do array_merge if static_write is enabled
            $eFile = \TYPO3\CMS\Core\Html\RteHtmlParser::evalWriteFile($vconf['spec']['static_write'], array_merge($currentRecord, $fieldArray));
        } else {
            $eFile = null;
        }
        // RTE transformations:
        if (!$this->dontProcessTransformations) {

I think even more easy optimizations are possible here (keep array_merge and do not create again in $mixedRec = array_merge($currentRecord, $fieldArray);) but this is only executed if static_write is enabled anyway.

For bulk insert of 6500 records:

  • 98000 calls less
  • 0.5 seconds faster
  • memory usage of array_merge()-Function down from 128MB to 6MB

Related issues 1 (0 open1 closed)

Related to TYPO3 Core - Task #63818: Remove staticFileEditPath magicClosed2014-12-12

Actions
Actions #1

Updated by Philipp Gampe almost 10 years ago

  • Parent task set to #63692
Actions #2

Updated by Gerrit Code Review almost 10 years ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35192

Actions #3

Updated by Gerrit Code Review almost 10 years ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35192

Actions #4

Updated by Gerrit Code Review almost 10 years ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35369

Actions #5

Updated by Gerrit Code Review almost 10 years ago

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35370

Actions #6

Updated by Gerrit Code Review almost 10 years ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35369

Actions #7

Updated by Christian Kuhn almost 10 years ago

the patch for master was abandoned, the issue is solved with #63818
for 6.2, the patch was re-pushed.

Actions #8

Updated by Gerrit Code Review almost 10 years ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35369

Actions #9

Updated by Stephan Großberndt almost 10 years ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100
Actions #10

Updated by Gerrit Code Review almost 10 years ago

  • Status changed from Resolved to Under Review

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35192

Actions #11

Updated by Christian Kuhn almost 10 years ago

  • Status changed from Under Review to Resolved
Actions #12

Updated by Benni Mack about 6 years ago

  • Status changed from Resolved to Closed
Actions

Also available in: Atom PDF