Project

General

Profile

Actions

Bug #63615

closed

Bug #63692: Memory consumption while bulk inserting

Huge memory consumption in DataHandler->processClearCacheQueue while bulk inserting

Added by Stephan Großberndt about 10 years ago. Updated about 6 years ago.

Status:
Closed
Priority:
Must have
Assignee:
-
Category:
Performance
Target version:
Start date:
2014-12-05
Due date:
% Done:

100%

Estimated time:
TYPO3 Version:
6.2
PHP Version:
Tags:
Complexity:
easy
Is Regression:
No
Sprint Focus:

Description

I found a memory consumption problem in

TYPO3\CMS\Core\DataHandling\DataHandler::processClearCacheQueue()
due to this dubious construct, which is called for each and every record added:

foreach (static::$recordsToClearCacheFor as $table => $uids) {
  foreach (array_unique($uids) as $uid) {
    ...

    foreach ($pageIdsThatNeedCacheFlush as $pageId) {
      if ($pageId >= 0) {
        $tagsToClear[] = 'pageId_' . $pageId;
      }
    }
    // Queue delete cache for current table and record
    $tagsToClear[] = $table;
    $tagsToClear[] = $table . '_' . $uid;
    $tagsToClear = array_unique($tagsToClear);
  }
}

I am adding 6500 records, so FOR EACH AND EVERY record 'pageId_' . $pageId, $table and $table . '_' . $uid are added to $tagsToClear and afterwards 'pageId_' . $pageId and $table are removed again for 6499 times. There is no reason to do it like this. array_unique() is meant here to reduce memory consumption in $tagsToClear but for big arrays with array_unique() this is terrible - xhprof showed 2,020,470,496 bytes (~2 gigabyte !!!) of used memory for array_unique() in processClearCacheQueue()

It is possible to set

clearCache_disable=1
to disable clearing of the cache but there is also a simple code solution to solve this since entries in this array are meant to be unique anyway:

foreach (static::$recordsToClearCacheFor as $table => $uids) {
  foreach (array_unique($uids) as $uid) {
    ...

    foreach ($pageIdsThatNeedCacheFlush as $pageId) {
      if ($pageId >= 0) {
        $tagsToClear['pageId_' . $pageId] = 'pageId_' . $pageId;
      }
    }
    // Queue delete cache for current table and record
    $tagsToClear[$table] = $table;
    $tagsToClear[$table . '_' . $uid] = $table . '_' . $uid;
  }
}

array_unique memory consumption drops from

IMemUse%: 13064.1%
Incl. MemUse (bytes): 2,020,470,496 bytes (~2 gigabyte !!!)

to
IMemUse%: 4.1%
Incl. MemUse (bytes): 629,464 bytes

There may be optimizations possible like

foreach (static::$recordsToClearCacheFor as $table => $uids) {
  foreach (array_unique($uids) as $uid) {
    ...

    foreach ($pageIdsThatNeedCacheFlush as $pageId) {
      $pageKey = 'pageId_' . $pageId;
      if ($pageId >= 0 && !isset($tagsToClear[$pageKey])) {
        $tagsToClear[$pageKey] = $pageKey;
      }
    }
    // Queue delete cache for current table and record
    if (!isset($tagsToClear[$table])) {
      $tagsToClear[$table] = $table;
    }
    $tagsToClear[$table . '_' . $uid] = $table . '_' . $uid;
  }
}

but even without those its SOOOO much better...

Regards,
Stephan

Actions

Also available in: Atom PDF