Bug #52235

Epic #55070: Workpackages

Epic #55065: WP: Overall System Performance (Backend and Frontend)

Bug #52949: Speed decrease since 4.5

Timeout when copying pages recursively due to cache-clearing overload

Added by Jan-Erik Revsbech about 6 years ago. Updated about 1 year ago.

Status:
Closed
Priority:
Must have
Assignee:
-
Category:
-
Target version:
Start date:
2013-09-24
Due date:
% Done:

100%

TYPO3 Version:
6.1
PHP Version:
Tags:
Complexity:
medium
Is Regression:
No
Sprint Focus:

Description

We currently experience a problem resulting in timeouts when copying more than a few pages, especially if the site is mounted over NFS. The problem seems to arise due to several issues.

The clear_cache command i DataHandler calls $GLOBALS['typo3CacheManager']->getCache('cache_pages')->flushByTag('pageId_' . $pageId); for each page it calculates should be cleared (including siblings, and possibly parents, grand-parents etc.).
Since this will ask all registered caches to clear for the desired tag, this results in many, many checks for flush caching. Combine this with a massively inefficient cache FileBackend (see #52125) this results in timeouts when copying more that 3-4 pages.

The result is even worse, since the clear_cache is called for the page being copied AND for each content element on the page, and for each content element the cache is cleared for its PID (if using workspaces, there is yet another problem with this, but that seems to be fixed in another ticket).

So copying a page with 4 content elements, will clear cache for the page 5 times! And for each of these, the siblings and possibly parents are cleared as well.

I would suggest a few changes and would like some feedback on this:

1: Make sure that only the cache_pages and possibly cache_pagesection is cleared by only calling $GLOBALS['typo3CacheManager']->getCache('cache_pages')->flushByTag('pageId_' . $pageId);
2: Make DataHandler collect a list of all pages it wants to clear, and then defer the actual clearing to later, to make sure we only clear cache one time for each page. We could either defer until the destructor of DataHandler is called, or we could just put everything
3: As suggested in #52125 FileBackend should either be fixed, or not used by default.

Feedback is welcome. I will push a changeset to gerrit with the suggested changes when possible feedback is collected.


Related issues

Related to TYPO3 Core - Bug #52125: Saving records takes ages to complete Closed 2013-09-19
Related to TYPO3 Core - Task #52295: Use SimpleFileBackend for t3lib_l10n cache Closed 2013-09-26
Related to TYPO3 Core - Bug #51116: Massive speed problem from TYPO3 version 6.x at the first page request after long period Closed 2013-08-16
Related to TYPO3 Core - Bug #61017: TCEMAIN.clearCacheCmd does not trigger Closed 2014-08-18
Duplicated by TYPO3 Core - Bug #57265: Importing large amounts of data issues many truncate's Rejected 2014-03-25

Associated revisions

Revision c9738f8c (diff)
Added by Jan-Erik Revsbech over 5 years ago

[BUGFIX] Fix problem with cache being cleared multiple time

When pages or records in general are changed using
the DataHandler, the pageCache is cleared for
each page multiple times. To fix this, the DataHandler
will collect which pages need clearing, and wait with
the actual clearing until all operations in
cmdMap or dataMap has been processed.
This way the cache is only cleared once for each page.

Resolves: #52235
Releases: 6.2
Change-Id: If1aa389991b801413eafa38d7d140f932bb321f2
Reviewed-on: https://review.typo3.org/25423
Reviewed-by: Markus Klein
Tested-by: Markus Klein
Reviewed-by: Jigal van Hemert
Reviewed-by: Jan-Erik Revsbech
Tested-by: Jan-Erik Revsbech
Tested-by: Philipp Gampe
Reviewed-by: Benjamin Mack
Tested-by: Benjamin Mack

History

#1 Updated by Markus Klein about 6 years ago

I'd suggest to implement some sort of batch cache clearing in the caches.
Example: * $GLOBALS['typo3CacheManager']->getCache('cache_pages')->setBatchFlushing(TRUE); * $GLOBALS['typo3CacheManager']->getCache('cache_pages')->flushByTag('pageId_' . 1); * $GLOBALS['typo3CacheManager']->getCache('cache_pages')->flushByTag('pageId_' . 2); * $GLOBALS['typo3CacheManager']->getCache('cache_pages')->flushByTag('pageId_' . 3); * ... * $GLOBALS['typo3CacheManager']->getCache('cache_pages')->setBatchFlushing(FALSE);

Once the batch flushing is disabled, the actual flushing will be executed.

#2 Updated by Jan-Erik Revsbech about 6 years ago

Markus Klein wrote:

I'd suggest to implement some sort of batch cache clearing in the caches.
Example:
  • $GLOBALS['typo3CacheManager']->getCache('cache_pages')->setBatchFlushing(TRUE);
  • $GLOBALS['typo3CacheManager']->getCache('cache_pages')->flushByTag('pageId_' . 1);
  • $GLOBALS['typo3CacheManager']->getCache('cache_pages')->flushByTag('pageId_' . 2);
  • $GLOBALS['typo3CacheManager']->getCache('cache_pages')->flushByTag('pageId_' . 3);
  • ...
  • $GLOBALS['typo3CacheManager']->getCache('cache_pages')->setBatchFlushing(FALSE);

Once the batch flushing is disabled, the actual flushing will be executed.

That is not a bad idea, but perhaps the DataHandler should be the one actually "queueing" things for cache_clearing? It could be done in the CachingLayer, but I'm not sure thats a good idea.

#3 Updated by Christian Kuhn about 6 years ago

just and idea on the 'batch' clearing (don't know if it works out): we have this "shutdown" method in bootstrap ... maybe we could execute the tag clearing there?

#4 Updated by Christian Kuhn about 6 years ago

  • Parent task set to #52304

#5 Updated by Markus Klein almost 6 years ago

  • Parent task changed from #52304 to #52949

#6 Updated by Ernesto Baschny over 5 years ago

This currently also makes importing a t3d file (used by the Distribution Management). Since we will see more people using this, we should try to tackle this issue soon.

Work in progress is here: https://review.typo3.org/#/c/25423/

#7 Updated by Philipp Gampe over 5 years ago

  • Status changed from New to Under Review

#8 Updated by Philipp Gampe over 5 years ago

  • Priority changed from Should have to Must have

#9 Updated by Gerrit Code Review over 5 years ago

Patch set 2 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/25423

#10 Updated by Gerrit Code Review over 5 years ago

Patch set 3 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/25423

#11 Updated by Gerrit Code Review over 5 years ago

Patch set 4 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/25423

#12 Updated by Gerrit Code Review over 5 years ago

Patch set 5 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/25423

#13 Updated by Gerrit Code Review over 5 years ago

Patch set 6 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/25423

#14 Updated by Gerrit Code Review over 5 years ago

Patch set 7 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/25423

#15 Updated by Gerrit Code Review over 5 years ago

Patch set 8 for branch master of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/25423

#16 Updated by Jan-Erik Revsbech over 5 years ago

  • Status changed from Under Review to Resolved
  • % Done changed from 0 to 100

#17 Updated by Markus Klein about 5 years ago

This caused a regression: #61017

#18 Updated by Benni Mack about 1 year ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF