Bug #98964
closed
Menu object caching creates too many records resulting in huge cache_hash table
100%
Description
Caching of menu objects causes excessive growth of the cache_hash table on many TYPO3 instances. In many cases, the table is filled with 5 to 10x the amount of records in the cache_pages, and on bigger sites the table can easily grow to several GB's (I have an instance where cache_hash grows to over 3GB, while cache_pages is still at a reasonable 350 to 400MB, with about 10.000 pages in cache). By far, most cache_hash records have an 'ident_MENUDATA' tag.
There are currently multiple problems with the caching of menu objects:¶
- the size of the tables can cause the server to run out of disk space
- many records of the created records are never retrieved, caching is pure overhead in this case (menu caching only plays a role at the first rendering of a page, as soon as the page is cached the menu caches are not used anymore for that page, provided you don't use uncached TS menu's...
- comparing the content of the cache records showed that many stored menu objects are identical, although the hash (field 'identifier') is different. At the mentioned site, SELECT count(distinct content) FROM cache_hash; resulted in 400 with 3200 records! So every object gets stored about 8 times in average.
Causes¶
- it appears that the calculated hash for storing the records has too many variables, causing identical menu objects to be stored. See $this->hash in AbstractMenuContentObject->makeMenu(). I'm not sure this can avoided with the current code.
- menus contain so many states (active, current, etc) that the output of most menus is different for every single page, making caching useless if the page itself is properly cached.
Possible solutions¶
- Benny Mack has created a much better alternative for the current menu objects with the extension b13/menus. This extension solves the issue by not storing all that state information in the menu cache, but adding that later on. This results in menu objects that can actually be re-used on many pages, so caching might actually result in some performance improvement, but of course still only for pages that are not in cache yet.
- I have very good experiences with skipping menu caching altogether, and propose to make it optional. Some quick and dirty benchmarking on my project showed a minor performance improvement on first page rendering, and - of course - no difference for cached pages. The large site where it is applied runs smooth as ever, with a very compact database now and no more disk space issues.
I will supply a patch for the second solution.
Updated by Loek Hilgersom over 1 year ago
- Related to Bug #20329: cache_hash fills up with huge amounts of MENUDATA added
Updated by Gerrit Code Review over 1 year ago
- Status changed from New to Under Review
Patch set 1 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/76384
Updated by Loek Hilgersom over 1 year ago
Important side-note:¶
This patch is a major improvement with the current state-full menus, where I think caching is a waste of time (making things more complicated, using more resources, and not reaching any performance improvement), because most menu objects are different for every page they are displayed.
But if the state-information is removed from the menus (like it is done in b13/menus), caching could lead to better performance. But not only that, if we ever want to tackle an issue like #14277 (Start/Stop time for pages is ignored in standard menu objects), we will need cached menus to also store a maximum TTL dependent on all pages possibly included in the menus.
Updated by Benni Mack over 1 year ago
- Related to Feature #14277: Start/Stop time for pages is ignored in standard menu objects added
Updated by Benni Mack over 1 year ago
- Related to Bug #91208: Performance issue in PageRepository for Mega Menus added
Updated by Benni Mack over 1 year ago
- Related to Bug #57953: Rendering time of HMENU is really bad, maybe bug? added
Updated by Sybille Peters over 1 year ago
In the review, it was suggested to drop the caching functionality altogether https://review.typo3.org/c/Packages/TYPO3.CMS/+/76384/1#message-78d64ddbcc50003315fe651516b609b0aa953923
since it does not make sense if the entries will be used only once.
If you DO have menus which are the same, e.g. a "target audience" menu or megamenu which is the same for all pages (of the same language), you can currently explicitly define the cache tag, e.g. with TypoScript using cache
So, IMHO it would be great to make sure this is still supported (in HMENU and MenuProcessor) for menus which will actually be reused. Because in that case, it can be a significant performance improvement.
Credit: Example reused from example David Bruchmann "Fully cached HMENU" (source)
lib.cachedMainMenu = COA
lib.cachedMainMenu.10 = HMENU
lib.cachedMainMenu.10 {
cache {
# use site identifier and language in cache key (menu is unique for every site / language)
key = cachedMainMenu-{site:identifier}-{siteLanguage:languageId}
key.insertData = 1
# default: use config.cache_period
lifetime = default
}
Updated by Sybille Peters over 1 year ago
- Related to Bug #98985: Property "cache" is not included in MenuProcessor added
Updated by Christian Kuhn over 1 year ago
Good point, Sybille.
".cache" TS property is a stdWrap feature and can be used "everywhere":
It does not interfere with the cache_hash call in AbstractMenuContentObject and shouldn't break when we change the cache strategy in the menu object code.
Updated by Gerrit Code Review over 1 year ago
Patch set 2 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/76384
Updated by Gerrit Code Review over 1 year ago
Patch set 3 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/76384
Updated by Gerrit Code Review over 1 year ago
Patch set 4 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/76384
Updated by Gerrit Code Review over 1 year ago
Patch set 5 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/76384
Updated by Gerrit Code Review over 1 year ago
Patch set 6 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/76384
Updated by Gerrit Code Review over 1 year ago
Patch set 7 for branch main of project Packages/TYPO3.CMS has been pushed to the review server.
It is available at https://review.typo3.org/c/Packages/TYPO3.CMS/+/76384
Updated by Loek Hilgersom over 1 year ago
- Status changed from Under Review to Resolved
- % Done changed from 0 to 100
Applied in changeset 5d22389b2240ad75ad0af423567d750d480c5bb9.