Bug #63615
closedBug #63692: Memory consumption while bulk inserting
Huge memory consumption in DataHandler->processClearCacheQueue while bulk inserting
100%
Description
I found a memory consumption problem in
TYPO3\CMS\Core\DataHandling\DataHandler::processClearCacheQueue()due to this dubious construct, which is called for each and every record added:
foreach (static::$recordsToClearCacheFor as $table => $uids) { foreach (array_unique($uids) as $uid) { ... foreach ($pageIdsThatNeedCacheFlush as $pageId) { if ($pageId >= 0) { $tagsToClear[] = 'pageId_' . $pageId; } } // Queue delete cache for current table and record $tagsToClear[] = $table; $tagsToClear[] = $table . '_' . $uid; $tagsToClear = array_unique($tagsToClear); } }
I am adding 6500 records, so FOR EACH AND EVERY record 'pageId_' . $pageId, $table and $table . '_' . $uid are added to $tagsToClear and afterwards 'pageId_' . $pageId and $table are removed again for 6499 times. There is no reason to do it like this. array_unique() is meant here to reduce memory consumption in $tagsToClear but for big arrays with array_unique() this is terrible - xhprof showed 2,020,470,496 bytes (~2 gigabyte !!!) of used memory for array_unique() in processClearCacheQueue()
It is possible to set
clearCache_disable=1to disable clearing of the cache but there is also a simple code solution to solve this since entries in this array are meant to be unique anyway:
foreach (static::$recordsToClearCacheFor as $table => $uids) { foreach (array_unique($uids) as $uid) { ... foreach ($pageIdsThatNeedCacheFlush as $pageId) { if ($pageId >= 0) { $tagsToClear['pageId_' . $pageId] = 'pageId_' . $pageId; } } // Queue delete cache for current table and record $tagsToClear[$table] = $table; $tagsToClear[$table . '_' . $uid] = $table . '_' . $uid; } }
array_unique memory consumption drops from
IMemUse%: 13064.1% Incl. MemUse (bytes): 2,020,470,496 bytes (~2 gigabyte !!!)
to
IMemUse%: 4.1% Incl. MemUse (bytes): 629,464 bytes
There may be optimizations possible like
foreach (static::$recordsToClearCacheFor as $table => $uids) { foreach (array_unique($uids) as $uid) { ... foreach ($pageIdsThatNeedCacheFlush as $pageId) { $pageKey = 'pageId_' . $pageId; if ($pageId >= 0 && !isset($tagsToClear[$pageKey])) { $tagsToClear[$pageKey] = $pageKey; } } // Queue delete cache for current table and record if (!isset($tagsToClear[$table])) { $tagsToClear[$table] = $table; } $tagsToClear[$table . '_' . $uid] = $table . '_' . $uid; } }
but even without those its SOOOO much better...
Regards,
Stephan
Updated by Stephan Großberndt about 10 years ago
This may be related: http://forum.typo3.org/index.php?t=msg&th=206887
Updated by Gerrit Code Review about 10 years ago
- Status changed from New to Under Review
Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35085
Updated by Gerrit Code Review about 10 years ago
Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35085
Updated by Gerrit Code Review about 10 years ago
Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35085
Updated by Gerrit Code Review about 10 years ago
Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35085
Updated by Gerrit Code Review about 10 years ago
Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35085
Updated by Lukas Krieger about 10 years ago
the gerrit issue-ID is wrong
Gerrit Code Review wrote:
Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35185
Updated by Stephan Großberndt about 10 years ago
This was an accidental commit to this issue. It should have been #63666 in the first place. It was fixed in the commit message and is shown in the correct issue now.
Stefan Neufeind corrected it here (strike-out and "wrong issue-ID"), but Benjamin Mack did another commit based on the old version. You can safely ignore it here. I requested another delete / strikeout in the typo3.teams.core mailinglist.
Updated by Alexander Opitz about 10 years ago
removed false commits here.
Last state is:
Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35085
Updated by Gerrit Code Review about 10 years ago
Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35211
Updated by Stephan Großberndt about 10 years ago
- Status changed from Under Review to Resolved
- % Done changed from 0 to 100
Applied in changeset 0f2f447e2ec834b69436bd5cca766353bf6d7024.
Updated by Gerrit Code Review about 10 years ago
- Status changed from Resolved to Under Review
Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35232
Updated by Stephan Großberndt about 10 years ago
- Status changed from Under Review to Resolved
Applied in changeset 6da6685887adf0386345d8c13ea092e70c3ace1b.