Bug #63668
closedBug #63692: Memory consumption while bulk inserting
High memory consumption in DataHandler->fillInFieldArray while bulk inserting
100%
Description
DataHandler->fillInFieldArray calls
$eFile = \TYPO3\CMS\Core\Html\RteHtmlParser::evalWriteFile($vconf['spec']['static_write'], array_merge($currentRecord, $fieldArray));
for each field of each record to insert. In each call array_merge($currentRecord, $fieldArray) is done only to throw it away again immediately and do nothing in most cases - does anyone use staticFileEditPath? I dont't think so: http://forum.typo3.org/index.php/t/195196/
$types_fieldConfig = BackendUtility::getTCAtypes($table, $currentRecord); $theTypeString = BackendUtility::getTCAtypeValue($table, $currentRecord); if (is_array($types_fieldConfig)) { foreach ($types_fieldConfig as $vconf) { // Write file configuration: // inserted array_merge($currentRecord,$fieldArray) 170502 $eFile = \TYPO3\CMS\Core\Html\RteHtmlParser::evalWriteFile($vconf['spec']['static_write'], array_merge($currentRecord, $fieldArray)); // RTE transformations: if (!$this->dontProcessTransformations) { }
Instead of unconditionally merging and throwing away again only merge if static_write is enabled
$types_fieldConfig = BackendUtility::getTCAtypes($table, $currentRecord); $theTypeString = BackendUtility::getTCAtypeValue($table, $currentRecord); if (is_array($types_fieldConfig)) { foreach ($types_fieldConfig as $vconf) { if (is_array($vconf['spec']['static_write']) && $GLOBALS['TYPO3_CONF_VARS']['BE']['staticFileEditPath']) { // only do array_merge if static_write is enabled $eFile = \TYPO3\CMS\Core\Html\RteHtmlParser::evalWriteFile($vconf['spec']['static_write'], array_merge($currentRecord, $fieldArray)); } else { $eFile = null; } // RTE transformations: if (!$this->dontProcessTransformations) {
I think even more easy optimizations are possible here (keep array_merge and do not create again in $mixedRec = array_merge($currentRecord, $fieldArray);) but this is only executed if static_write is enabled anyway.
For bulk insert of 6500 records:
- 98000 calls less
- 0.5 seconds faster
- memory usage of array_merge()-Function down from 128MB to 6MB
Updated by Gerrit Code Review almost 10 years ago
- Status changed from New to Under Review
Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35192
Updated by Gerrit Code Review almost 10 years ago
Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35192
Updated by Gerrit Code Review almost 10 years ago
Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35369
Updated by Gerrit Code Review almost 10 years ago
Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35370
Updated by Gerrit Code Review almost 10 years ago
Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35369
Updated by Christian Kuhn almost 10 years ago
the patch for master was abandoned, the issue is solved with #63818
for 6.2, the patch was re-pushed.
Updated by Gerrit Code Review almost 10 years ago
Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35369
Updated by Stephan Großberndt almost 10 years ago
- Status changed from Under Review to Resolved
- % Done changed from 0 to 100
Applied in changeset c1965e2166e580a68d539aa4382f4d3367035d8d.
Updated by Gerrit Code Review almost 10 years ago
- Status changed from Resolved to Under Review
Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35192
Updated by Christian Kuhn almost 10 years ago
- Status changed from Under Review to Resolved
Applied in changeset 4dd99fee1f24f4106a153743fb59f98dbae268b5.