Project

General

Profile

Actions

Bug #63615

closed

Bug #63692: Memory consumption while bulk inserting

Huge memory consumption in DataHandler->processClearCacheQueue while bulk inserting

Added by Stephan Großberndt about 9 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Must have
Assignee:
-
Category:
Performance
Target version:
Start date:
2014-12-05
Due date:
% Done:

100%

Estimated time:
TYPO3 Version:
6.2
PHP Version:
Tags:
Complexity:
easy
Is Regression:
No
Sprint Focus:

Description

I found a memory consumption problem in

TYPO3\CMS\Core\DataHandling\DataHandler::processClearCacheQueue()
due to this dubious construct, which is called for each and every record added:

foreach (static::$recordsToClearCacheFor as $table => $uids) {
  foreach (array_unique($uids) as $uid) {
    ...

    foreach ($pageIdsThatNeedCacheFlush as $pageId) {
      if ($pageId >= 0) {
        $tagsToClear[] = 'pageId_' . $pageId;
      }
    }
    // Queue delete cache for current table and record
    $tagsToClear[] = $table;
    $tagsToClear[] = $table . '_' . $uid;
    $tagsToClear = array_unique($tagsToClear);
  }
}

I am adding 6500 records, so FOR EACH AND EVERY record 'pageId_' . $pageId, $table and $table . '_' . $uid are added to $tagsToClear and afterwards 'pageId_' . $pageId and $table are removed again for 6499 times. There is no reason to do it like this. array_unique() is meant here to reduce memory consumption in $tagsToClear but for big arrays with array_unique() this is terrible - xhprof showed 2,020,470,496 bytes (~2 gigabyte !!!) of used memory for array_unique() in processClearCacheQueue()

It is possible to set

clearCache_disable=1
to disable clearing of the cache but there is also a simple code solution to solve this since entries in this array are meant to be unique anyway:

foreach (static::$recordsToClearCacheFor as $table => $uids) {
  foreach (array_unique($uids) as $uid) {
    ...

    foreach ($pageIdsThatNeedCacheFlush as $pageId) {
      if ($pageId >= 0) {
        $tagsToClear['pageId_' . $pageId] = 'pageId_' . $pageId;
      }
    }
    // Queue delete cache for current table and record
    $tagsToClear[$table] = $table;
    $tagsToClear[$table . '_' . $uid] = $table . '_' . $uid;
  }
}

array_unique memory consumption drops from

IMemUse%: 13064.1%
Incl. MemUse (bytes): 2,020,470,496 bytes (~2 gigabyte !!!)

to
IMemUse%: 4.1%
Incl. MemUse (bytes): 629,464 bytes

There may be optimizations possible like

foreach (static::$recordsToClearCacheFor as $table => $uids) {
  foreach (array_unique($uids) as $uid) {
    ...

    foreach ($pageIdsThatNeedCacheFlush as $pageId) {
      $pageKey = 'pageId_' . $pageId;
      if ($pageId >= 0 && !isset($tagsToClear[$pageKey])) {
        $tagsToClear[$pageKey] = $pageKey;
      }
    }
    // Queue delete cache for current table and record
    if (!isset($tagsToClear[$table])) {
      $tagsToClear[$table] = $table;
    }
    $tagsToClear[$table . '_' . $uid] = $table . '_' . $uid;
  }
}

but even without those its SOOOO much better...

Regards,
Stephan

Actions #2

Updated by Gerrit Code Review about 9 years ago

  • Status changed from New to Under Review

Patch set 1 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35085

Actions #3

Updated by Gerrit Code Review about 9 years ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35085

Actions #4

Updated by Gerrit Code Review about 9 years ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35085

Actions #5

Updated by Gerrit Code Review about 9 years ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35085

Actions #6

Updated by Philipp Gampe about 9 years ago

  • Parent task set to #63692
Actions #7

Updated by Gerrit Code Review about 9 years ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35085

Actions #8

Updated by Gerrit Code Review about 9 years ago

false commit removed

Actions #9

Updated by Gerrit Code Review about 9 years ago

false commit removed

Actions #10

Updated by Lukas Krieger about 9 years ago

the gerrit issue-ID is wrong

Gerrit Code Review wrote:

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35185

Actions #11

Updated by Stephan Großberndt about 9 years ago

This was an accidental commit to this issue. It should have been #63666 in the first place. It was fixed in the commit message and is shown in the correct issue now.

Stefan Neufeind corrected it here (strike-out and "wrong issue-ID"), but Benjamin Mack did another commit based on the old version. You can safely ignore it here. I requested another delete / strikeout in the typo3.teams.core mailinglist.

Actions #12

Updated by Alexander Opitz about 9 years ago

removed false commits here.

Last state is:

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35085

Actions #13

Updated by Gerrit Code Review about 9 years ago

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35211

Actions #14

Updated by Stephan Großberndt about 9 years ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100
Actions #15

Updated by Gerrit Code Review about 9 years ago

  • Status changed from Resolved to Under Review

Patch set 1 for branch TYPO3_6-2 of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at http://review.typo3.org/35232

Actions #16

Updated by Stephan Großberndt about 9 years ago

  • Status changed from Under Review to Resolved
Actions #17

Updated by Benni Mack over 5 years ago

  • Status changed from Resolved to Closed
Actions

Also available in: Atom PDF