Bug #57953
closedRendering time of HMENU is really bad, maybe bug?
0%
Description
We use the following menu on every page.
Because of expAll = 1 on the first two levels, the menu is by far the slowest element on the pages. This Bug has been adressed in Ticket (http://forge.typo3.org/issues/23486) so far, but was closed due to missing feedback.
Also I am unsure whether this is actually the problem, but in combination with two other menus on the page it is responsible for 90% of the processing time.
60 = HMENU 60 { entryLevel = 1 1 = TMENU 1 { expAll = 1 NO.allWrap = <li>|</li> NO.ATagTitle.field = abstract // description // title ACT = 1 ACT.wrapItemAndSub = <li class="active">|</li> ACT.ATagTitle.field = abstract // description // title IFSUB = 1 IFSUB.ATagTitle.field = abstract // description // title IFSUB.stdWrap.wrap = | <span class="text-muted">/ ...</span> IFSUB.stdWrap2.wrap = |<ul class="nav collapse"> IFSUB.wrapItemAndSub ( <li class="sub"> <a href="#" title="Untermenü aufklappen" class="sidebar-dropdown-btn" data-toggle="collapse" data-parent="#sidebar .panel-body.nav"> <span class="caret"></span> </a> |</ul> </li> ) ACTIFSUB < .IFSUB ACTIFSUB.stdWrap2.wrap = |<ul class="nav collapse in"> ACTIFSUB.wrapItemAndSub ( <li class="active sub"> <a href="#" title="Untermenü aufklappen" class="sidebar-dropdown-btn" data-toggle="collapse" data-parent="#sidebar .panel-body.nav"> <span class="caret"></span> </a> |</ul> </li> ) CURIFSUB < .ACTIFSUB wrap = <ul id="sidebar-menu" class="panel-body nav">|</ul> } 2 < .1 2.wrap > 3 = TMENU 3 { NO.allWrap = <li>|</li> NO.ATagTitle.field = abstract // description // title ACT = 1 ACT.wrapItemAndSub = <li class="active">|</li> ACT.ATagTitle.field = abstract // description // title ACTIFSUB < .ACT CURIFSUB < .ACT } wrap = <div id="sidebar-menu" class="panel-collapse collapse in">|</div> }
Files
Updated by Alexander Stehlik over 10 years ago
Thanks for bringing this up. I also have the impression that menu rendering takes quite long. I also started looking around a bit but I did not find a bottleneck. The process simply seems to be complex (access checking, translation checking etc.).
But it would be really cool if someone has an idea. Especially for rendering sitemaps the slow menu rendering is a real pain.
Updated by Florian Schwaiger over 10 years ago
- File PHPinfo_Localhost.html PHPinfo_Localhost.html added
- File PHPinfo_Staging.html PHPinfo_Staging.html added
- File Status_Localhost.txt Status_Localhost.txt added
- File Status_Staging.txt Status_Staging.txt added
I compared my two servers and did not find a clue to the difference in performance yet (that is, the local machine takes 40ms to render, the remote one 4000ms). Find attached the server configs and mysql table status, perhaps you can spot a difference.
The only other thing I can imagine, that the mysql server on my local machine is a lot faster due to the SSD disk.
Updated by Florian Schwaiger over 10 years ago
In case someone has more experience analyzing profiler data, I conducted 3 test cases using the xdebug profiler:
0) without profiler, cleared FE caches
0) without profiler, reloaded news list to fill general caches
1) with profiler, loaded single news object
2) with profiler, loaded other single news object
3) with profiler, loaded calendar page
1) https://dl.dropboxusercontent.com/u/70926418/xdebug.local.out.1
2) https://dl.dropboxusercontent.com/u/70926418/xdebug.local.out.2
3) https://dl.dropboxusercontent.com/u/70926418/xdebug.local.out.3
1) https://dl.dropboxusercontent.com/u/70926418/xdebug.staging.out.1
2) https://dl.dropboxusercontent.com/u/70926418/xdebug.staging.out.2
3) https://dl.dropboxusercontent.com/u/70926418/xdebug.staging.out.3
Updated by Florian Schwaiger over 10 years ago
So, we found out that indeed the performance problem has to do with too much database access. Finally, after searching for hours, I almost accidentally looked into the configuration module and found out, that even though apc was enabled in the install tool, it was not used anywhere.
Adding the following lines to typo3conf/AdditionalConfiguration.php disabled most cache to the database and now page rendering times dropped to a 10th! This is now acceptable.
$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_pages']['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\ApcBackend'; $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_pages']['options'] = array(); $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_hash']['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\ApcBackend'; $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_hash']['options'] = array(); $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_pagesection']['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\ApcBackend'; $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_pagesection']['options'] = array(); $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_rootline']['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\ApcBackend'; $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_rootline']['options'] = array(); $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['extbase_reflection']['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\ApcBackend'; $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['extbase_reflection']['options'] = array();
The question remains though, why does enabling the APC backend in the install tool not affect TYPO3 default caching behavior?
Updated by Alexander Stehlik over 10 years ago
Hi Florian,
thank you very much for your research.
Where do you set the APC cache in the install tool? I know one setting under "Configuration Presets" but this only changes the cache type for the extbase_object
cache.
I'm not sure which of your caches improves the menu rendering performance. Or does it only improve the performance for cached pages?
As far as I can see most of the menu rendering is done in the \TYPO3\CMS\Frontend\Page\PageRepository
(e.g. ->getMenu()
) which only uses direct database queries or its local caches. So I can not see how changing the configuration for all the other caches can improve the rendering times for menus.
Cheers,
Alex
Updated by Alexander Stehlik over 10 years ago
Another thought: I'm wondering if the performance could be improved using MySQL indexes for the fields queried by the PageRepository, e.g. deleted, hidden, starttime, endtime, fe_group and optionally the versioning fields.
Updated by Florian Schwaiger over 10 years ago
Where do you set the APC cache in the install tool? I know one setting under "Configuration Presets" but this only changes the cache type for the extbase_object cache.
I set it in the install tool only, it only changed extbase_object cache. Now I have this file in AdditionalConfiguration as I see no benefit at all for storing anything in database:
if (!function_exists('apc_for_cache_backend')) { function apc_for_cache_backend($cacheName) { $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations'][$cacheName]['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\ApcBackend'; $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations'][$cacheName]['options'] = array(); } } apc_for_cache_backend('cache_pages'); apc_for_cache_backend('cache_hash'); apc_for_cache_backend('cache_pagesection'); apc_for_cache_backend('cache_rootline'); apc_for_cache_backend('extbase_reflection'); apc_for_cache_backend('cache_news_category');
I'm not sure which of your caches improves the menu rendering performance. Or does it only improve the performance for cached pages?
I checked on cache performance once more by resetting the "cache_hash" table to database backend.
The rendering times increased to 4 seconds once more, so I believe the bottleneck is with this cache table.
Updated by Jörg Wagner about 10 years ago
I can fully confirm this report on TYPO3 CMS 6.2.4.
Some more details to reproduce:- The effect is NOT visible if your DB is running from an SSD, so you need a classical HD based MySQL server to reproduce
- expAll = 1 must be used to force the rendering of the complete menu tree
- you don't need a very big tree but you need some nesting, and ... very important ...
- the MenuItem depth must be reasonably big (e.g. 2 < .1, 3 < .1, 4 < .1 - see example below).
Here is a quick sketch of a minimal HMENU to get the effect.
page.10 = HMENU page.10 { 1 = TMENU 1 { expAll = 1 wrap = <ul>|</ul> NO = 1 NO { wrapItemAndSub = <li>|</li> } } 2 < .1 3 < .1 4 < .1 }
With a small page tree of some 30 pages (total page count over all levels) nested three levels deep, I get rendering times of 10 to 15 seconds per page (with pages that contain nothing but the above menu)! After a page has been rendered once and has been stored in cache, the next call serves the page reasonably fast. But with only 30 pages you can easily calculate that the total rendering time for all pages on first hit sums up to a total of 5 minutes. This is beyond any reasonable figure, especially considering that the pages are practically empty (only a menu contained).
I have tested all of this on two server systems with reasonable performance, one of them being a DomainFactory (Dedicated Server, nearly empty) which has always been a great and speedy hoster for us.
My testing CMS is a fresh and empty TYPO3 installation. On DomainFactory I have even used their TYPO3 default installation to test.
My conclusion is that there is some major problem in the back-ported caching framework. As a side note, I also have rendering times of about 15 seconds for a simple Extbase phone list extension in another project. That phone list has only one person table with 600 records and two 1:n tables for person attributes (company and department, about 20 records each). My profiling trials lead to the same conclusion: The caching framework eats up unbelievable resources.
Considering the ramifications of both effects (menu performance problems and extbase extension performance problems) and both issues obviously originating from the caching framework, I have to conclude, that 6.2 is simply not yet production ready. Unfortunately the problem is buried so deep in the caching system that I feel unable to debug it myself and present a solution (as I have done so many times before for TYPO3 over the last 10 years).
Updated by Nikola Vignjevic almost 10 years ago
Hi everyone.
I have here the same problem.
I have the 6.2.4 version with the demo pages from the bootstrap_package. I watch the rendering time via admintool, and the results are this:
I take this page (not cashed) as example:
/about/history.html
The HMENU is called 6 times, (that seems OK) but this times:
10 HMENU 1364 1493 +12 =1505
10 HMENU 2870 653 +17 =670
10 HMENU 3541 1174
10 HMENU 4779 891
20 HMENU 5704 3381
10 HMENU 9089 148
Only the HMENU costed here 7.611 sec.
The times are every time a different, but nearly every time the HMENU costs between 50% and 75% of the whole rendering time.
I was now over one week looking for the bottleneck, and fond two points where the speed is going away in typo3.
The second was in the view, if you have a loops or similar elements, which you use to build for example tables, edit forms or something like that, for example:
<f:for each="{stores}" as="store">
... here some needful table rows ...
</f:for>
If you have both things HMENU and a plugin, which use f:for you get loading times up to:
-->> 15 sec.
maybe on servers with a better setup, SSHDs and with workarounds You get some better results. but every page with a simple navigation and a table with 20 rows which need longer then 0.3 - 0.5 sec is in my opinion google-dead; and that is for me a K.O. criterion ,... :-(
please help.
best regards
Nik
Updated by Jan Kiesewetter over 9 years ago
I can confirm this problem.
I have a website with about 120 pages on 4 levels.
The rendering of an uncached page takes 5000ms, the HMENU takes 3500ms of them, so about 70%.
TYPO3 6.2.6
Updated by Elmar Putz over 9 years ago
I have the same problem. On Typo3 6.2.9, a website with about 150 pages on 4 levels is very slow rendering the uncached pages.
When deactivating the menus, page rendering time improves about 3000%. When page is cached, performance is ok, but the initial loading takes endlessly.
For testing purposes i created a special page type, which returns the menu (4 levels) only - it takes about 5 sec. to render on a hetzner managed server q90, which is a quite powerful machine. Other Typo3 projects on the same machine with a similar number of pages, but less menu levels are delievered in a blink of an eye
Updated by Matthew Colton over 9 years ago
I can confirm this problem.
On Typo3 6.2.9, on a site with about 150 pages and a three level Menu HMENU rendering takes about 12 seconds according to the profiler extension. The configuration is simuliar to the other examples on this page.
Once the page is cached the performance is ok. The server is a root server with a very low load running PHP 5.5.19 with memory_limit set to 256M.
Updated by Alexander Opitz over 9 years ago
- Category set to Performance
Would like to take a look into this over the christmas days.
Does someone have a demonstrational setup and can send me the data so I only need to import all into my testing system?
Updated by Jörg Wagner over 9 years ago
Hi Alexander,
all you need is:
– a page tree with some 30 empty pages nested 4 to 5 levels deep (quick to build with the Functions module and some page dragging to nest the pages)
– a HMENU of this form:
page.10 = HMENU page.10 { 1 = TMENU 1 { expAll = 1 wrap = <ul>|</ul> NO = 1 NO { wrapItemAndSub = <li>|</li> } } 2 < .1 3 < .1 4 < .1 5 < .1 }
Make sure that you do NOT host your MySQL database on an SSD, as this will increase the db access speed so much that you will not witness the problem.
If you use a Mac Book Pro with an SSD system drive, you must temporarily move your MySQL storage to an externally attached conventional HD.
Thanks for looking into this!
Cheers, Jörg.
Updated by Alexander Opitz over 9 years ago
SSD or not is not the question. I will take a look into all SQL Queries that will be executed.
Updated by Andre Michels over 9 years ago
I upgraded one installation of Typo3 6.2.7 to Typo3 7 because of this problem.
It looks like the bug was fixed. Probably we only need a back port of the chaching changes?
Updated by Florian no-lastname-given over 9 years ago
I have the same problem with Typo3 6.2.9. Rendering of my main menu based on HMENU takes more than 10 seconds.
APC saved my day (see above) and rendering performance is ok now.
If I can help anyone fixing this problem feel free to contact me.
Merry Christmas
Florian
Updated by Arjen Hoekema over 9 years ago
Could be related to: https://forge.typo3.org/issues/63550
Ran into some performance issues with a TMENU using expAll = 1 where empty menu objects were stored in cache (cache_hash) as NULL. The cache was checked using 'is_array' so the menu object was regenerated.
On some pages generating the page took 5 sec, now dropped to < 1s.
Updated by Arjen Hoekema over 9 years ago
Also check if $GLOBALS['TYPO3_CONF_VARS']['FE']['activateContentAdapter'] is still activated which results in providing a fallback for the 'media' field from the 'pages' table and generating numerous additional queries for menu items.
Updated by Jörg Wagner over 9 years ago
@Arjen Hoekema:
A) #63550 sounds very promising. This could very well be the source of the issue here!
B) ['FE']['activateContentAdapter'] is FALSE in all of our environments. This was not the reason for the problem in our tests.
@Florian Kuss:
This has nothing to do with APC or any other OPCache system. We use OPCaches in all our live systems and still have the problem.
Updated by Sebastian Lemke over 9 years ago
Hello,
is it sure that caching helps out here? Sure - when the cache works correcty (#63550) it will be much faster.
But I do not understand why generating the MENU itself takes such a long time, regardless of the cache?
Sebastian
Updated by Stefan Neufeind over 9 years ago
Depending on the number of pages a huge number of SQL-queries are fired. Last time I looked there was a query (or two?) for every page being displayed in the menu. That might be related to checks for visibility or so.
Updated by Andre Michels over 9 years ago
I've applied the #63550 patch but clearly it doesn't fix the problem at all. This bug is very annoying in our bigger Typo3 instances. Looked serveral hours, debugging, writing emails to customers... I don't understand why this bug doesn't affect so much people. In Typo3 7 the bug was fixed, but where?
Updated by Alexander Stehlik over 9 years ago
I think not everyone is affected by this bug because it heavily depends on your database / IO performance.
When your storage performance is not optimal the impact of this problem seems to be much bigger.
When you have fast harddisks (SSDs) you will not notice this problem so much.
At least these are my experiences.
Updated by Andre Michels over 9 years ago
We run MariaDB on huge DB servers, they arent slow... But the slowquery log shows very slow querys (1-5s) for INSERT statements in the cf_cache_hash.
Updated by Mario Hirschfeld over 9 years ago
confirmed
Actually we´re updating round about 30 Typo3-Services from 4.5 to 6.2 LTS but can´t explain these performance problems to our customers.
A pulldownmenu with 250 entries in 3 menulevel will slow down the service to 20 seconds response time without any content on the site.
This is a Root-Server with SSDs.
A sulution is needed fast as possible, we can´t work with typo3 6 since it´s fixed.
Updated by Alexander Stehlik over 9 years ago
We are using a redis cache as a workaround. This configuration improved the speed for us quite a bit:
if ( isset($GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['myext']['enableRedisCaches']) && $GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['myext']['enableRedisCaches'] && extension_loaded('redis') ) { if (!function_exists('configureRedisAsCacheBackend')) { function configureRedisAsCacheBackend($cacheName) { $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations'][$cacheName]['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\RedisBackend'; $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations'][$cacheName]['options'] = array(); } } configureRedisAsCacheBackend('cache_pages'); configureRedisAsCacheBackend('cache_hash'); configureRedisAsCacheBackend('cache_pagesection'); configureRedisAsCacheBackend('cache_rootline'); configureRedisAsCacheBackend('extbase_reflection'); configureRedisAsCacheBackend('cache_news_category'); }
Updated by Clemens Tilgner over 9 years ago
We skippt the Menu generation with Typoscript and used a small Fluid Template to generate the Menu. This helped speeding up the rendering process a lot.
It is a workaround, but no direct solution for this problem.
Updated by Jörg Wagner over 9 years ago
Clemens Tilgner wrote:
We skippt the Menu generation with Typoscript and used a small Fluid Template to generate the Menu. This helped speeding up the rendering process a lot.
It is a workaround, but no direct solution for this problem.
How do you do that? VHS viewhelper?
Updated by Stefan Neufeind over 9 years ago
If the menu is the same on all pages, you can cache it so it will only be rendererd on the first accessed page (taking the performance-penalty) and later be fetched from cache for all other pages that are generated.
One possibility for caching is for example stdWrap.cache: http://docs.typo3.org/typo3cms/TyposcriptReference/Functions/Cache/Index.html
Updated by Clemens Tilgner over 9 years ago
Jörg Wagner wrote:
How do you do that? VHS viewhelper?
Correct. This is the template we used:
page.10.marks.NAVI = FLUIDTEMPLATE
page.10.marks.NAVI {
file = fileadmin/templates/fluid-templates/mainmenu.html
}
mainmenu.html: {namespace v=FluidTYPO3\Vhs\ViewHelpers}
<v:page.menu pageUid="1" expandAll="1" useShortcutUid="0">
<ul id="nav">
<f:for each="{menu}" as="menuPage">
...
</f:for>
</ul>
</v:page.menu>
Updated by Faktor E GmbH over 9 years ago
Any update for that issue? It really is a pain right now for some of our installations with 2 or more levels in the HMENU. We have 6.2.11 running.
Updated by taywa gmbh about 9 years ago
We can confirm this issue with 6.2.11 which includes #63550. on a large site with hundreds of pages the generation of an empty page with just the menu takes over a minute.
it is considerably better with 6.2.12 but it is still slow, about ~12 seconds. 4.5 LTS was able to do that in ~3 seconds. (tested with the same page, one upgraded one old, on the same server on two different ports)
Updated by Tao Nguyen about 9 years ago
Hi, we have copied our Typo3 4.5 to a test environment to Typo3 6.2.13. We use a nested HMenu (3 levels) and about 120 Pages. Before 6.2.12 it took about 20-30 seconds minimum to load a page without cache, with 6.2.13 it is now much better and takes about 7-12 seconds to load completely. We tried to optimize the TS as much as possible, removing conditions (except language conditions) and using less extensions. I even removed the ACT-state from the HMENU (so the menu basically always stays the same and is supposed to load very quick).
Everything else loads quick if I take a look at the adminpanel, only the language HMENU and the navigation HMENU take 1-2 seconds and 5-11 seconds respectively. The more or less same installation on 4.5 loads without cache in about 3 seconds.
I initially tried to convince my client to update to 6.2 because it is the new LTS with all its benefits, but the initial loading time is not acceptible for him (at least it is still quick with static cache after the first loading). Is the only solution putting the database to a SSD?
Updated by Jörg Wagner about 9 years ago
Is the only solution putting the database to a SSD?
As mentioned above there is also the option to use VHS viewhelper.
But our solution for now is that we have moved all database storage files to SSDs on our public servers. Luckily this is possible with our hoster at reasonable cost. This completely irons out the performance issues of TYPO3 6.2 for all our hosted customers and cured a lot of headaches.
Anyway, it is quite embarrassing that such measures are still necessary 15 months after launch of 6.2.
Updated by Tao Nguyen about 9 years ago
Jörg Wagner wrote:
Is the only solution putting the database to a SSD?
As mentioned above there is also the option to use VHS viewhelper.
Hi Jörg, thanks for the tip, It took me awhile to be able to create a 3 level columnar dropdown menu. Here is my code, if it might help someone. I had to adapt the css since I couldn't find out how to adapt Typoscript's wrapItemAndSub correctly with fluid. The Website loads faster now, 6-8 seconds and without spikes (with HMENU it sometimes took anything between 12 and 50 seconds).
{namespace v=FluidTYPO3\vhs\ViewHelpers} <v:page.menu useShortcutData="TRUE" pageUid="264" expandAll="1" levels="3"> <ul class="menu"> <f:for each="{menu}" as="item"> <li> <f:link.page pageUid="{item.uid}" title="{item.linktext}">{item.linktext}</f:link.page> <f:if condition="{item.hasSubPages}"> <v:page.menu pageUid="{item.uid}"> <ul class="second"> <f:for each="{menu}" as="subItem"> <li> <f:link.page pageUid="{subItem.uid}" title="{subItem.linktext}">{subItem.linktext}</f:link.page> <f:if condition="{item.hasSubPages}"> <v:page.menu pageUid="{subItem.uid}"> <ul class="third"> <f:for each="{menu}" as="subsubItem"> <li> <f:link.page pageUid="{subsubItem.uid}" title="{subsubItem.linktext}">{subsubItem.linktext}</f:link.page> </li> </f:for> </ul> </v:page.menu> </f:if> </li> </f:for> </ul> </v:page.menu> </f:if> </li> </f:for> </ul> </v:page.menu>
Updated by Simon Schaufelberger about 9 years ago
Instead of nesting all the levels, a main template with section would be helpful in this case which calls itself. Something like this (from Neos):
Updated by Philipp Kerling almost 9 years ago
You don't need to repeat the menu for every level, just use <v:page.menu.sub>
Updated by Riccardo De Contardi almost 7 years ago
- File Cattura_a_1500pages.PNG Cattura_a_1500pages.PNG added
- File Cattura_a_3000pages.PNG Cattura_a_3000pages.PNG added
- File Cattura_a_6000pages.PNG Cattura_a_6000pages.PNG added
- File Cattura_a_9000pages.PNG Cattura_a_9000pages.PNG added
- Status changed from New to Needs Feedback
I've performed a test on a fresh 8.7.6 with the introduction package under the following conditions:
1) Environment: Xampp server on my Windows7 laptop (Intel Core I5 with 6GB ram, NO SSD disk); php 7.0 with:
max_input_time 60 max_input_vars 1500 memory_limit 1024M
2) Typoscript setup: the page contains just a 4-level menu and cache is disabled
config.no_cache=1 page > page=PAGE page.10 = HMENU page.10 { 1 = TMENU 1 { expAll = 1 wrap = <ul>|</ul> NO = 1 NO { wrapItemAndSub = <li>|</li> } } 2 < .1 3 < .1 4 < .1 }
3) Test: I've started adding a new page with 20 subpages per level each (I.e 20 pages at first sublevel, and each page has 20 subpages and so forth till the 4th level).
Even though I'm far from finished (I started with the intention of adding 20 pages at the first level with a complete 4-level tree of 20pages each), I could already share some results; I've taken screenshots of the admin panel
with various amounts of pages added.
I've had to add about 9000+ pages before the rendering time goes near 20 seconds.
In the latter condition (9000+ pages), with cache enabled and filled, the rendering time is about 300ms
Do you think I could perform some other tests? Or can I consider the issue solved? Thank you
Updated by Julian Hofmann over 6 years ago
Beside the solutions with fluid or use of APC, there's another option to speedup big HMENUs with expAll: use the possibility of caching.
Normally, big parts of these menus does not change on each page, but the whole HMENU is rendered per page. Why not using identical parts of the menu for different pages? All inactive branches of the menu could be cached. This lowers the count of database queries.
For details of such a solution, there's a blogpost: http://stichwort.org/typo3/typoscript/mega-menue-mit-cache-optimieren/ (German only)
Updated by Riccardo De Contardi over 6 years ago
- Status changed from Needs Feedback to Closed
I think I can close this one for now, as there has been no activity since the last 4 months, and this perfomance issue
should be solved now for versions 7 and 8 (moreover, here a couple of possible strategies have been suggested to reduce the loading times).
If you think that this is the wrong decision or experience the issue again or think that there is still work to be done about this topic, please reopen it or open a new issue with a reference to this one.
Updated by Jonas Ulrich about 6 years ago
We've written an alternative, based on the addition to the blog post of the last comment, made on 2018-05-17:
Addition in the blogpost: https://gist.github.com/pgampe/cb29bc0fc1111d1370cc
Our generalized approach, based on that: https://gist.github.com/julrich/d91c1ac1968418e6ae8290f2a2e90afb
Maybe this is helpful for somebody else!
Handles cases like the special pages (index page, search page, pages not inside the root node of the menu, where the replacement RegExp otherwise would be an empty string, replacing everything), logged in users and languages.
Btw. what make you say this: "should be solved now for versions 7 and 8", Riccardo? At least for us, it wasn't (8.7).
Updated by Riccardo De Contardi about 6 years ago
@Jonas Ulrich basically there has not been much activity on this issue; I guessed it was solved because of the lack of people complaining about it... It it absolutely possible that I am wrong. I also tried an empiric test 9 months ago and had no feedback...
Please, if you have more information about how to reproduce the problem write it; I'll reopen it if you want.
Regards,
Riccardo
Updated by Jonas Ulrich about 6 years ago
- File profiling-hmenu.png profiling-hmenu.png added
Not sure if there is much to do about this, to be honest.
It's just expAll in a HMENU resulting in a lot of SQL calls, and corresponding TYPO3 function calls (render, makeMenu, prepareMenuItems, getSubpagesForPages, etc).
About 120% of the number of pages in SQL calls (e.g. 1000 pages, ~1200 SQL requests).
Approximately 1/8th is spent in actually executing the SQL queries (in a reference profile 841ms (1232 SQL requests) of 6.26s.
The rest (7/8) ist mostly in those mentioned calls.
I've attached a sample profiling, outlining those calls. Sorry about the quality, but had to quickly stitch this together from multiple screenshots:
Updated by Markus Klein about 6 years ago
@Jonas
1) Please edit your message and use the thumbnail feature to avoid this huge image inline.
2) Please add the corresponding typoscript which generated this graph.
Thanks.
Updated by Jonas Ulrich about 6 years ago
I've changed the image to use the thumbnail, sorry about that.
As our menu is quite complex, I can't really rip it out (multiple .ts-files) easily.
I will see (can't promise anything) if I can add a reduced example. But to be honest, to me it seems that the problem is just unchanged.
I'd hazard a guess, but the minimal example, posted by Jörg Wagner 3 years ago, should also still exhibit those performance penalties for large(r) menus:
https://forge.typo3.org/projects/typo3cms-core/activity?from=2014-12-17
Updated by Jarvis H over 5 years ago
- Related to Bug #87729: MenuProcessor can't cache added
Updated by Benni Mack over 4 years ago
- Status changed from New to Needs Feedback
Please see https://github.com/b13/menus if this solves your issue.
Our cache entries went down MASSIVELY by not relying on HMENU anymore.
Updated by Stefan Froemken over 4 years ago
- Related to Bug #91208: Performance issue in PageRepository for Mega Menus added
Updated by Sybille Peters over 3 years ago
- Tags set to large-site,performance,menu
Updated by Sybille Peters over 3 years ago
- Related to Epic #93547: Collection of problems with large sites added
Updated by Christian Kuhn almost 3 years ago
- Status changed from Needs Feedback to Closed
Hey. I'm gonna close this for now: It is known that menu rendering suffers from RootlineUtility "tree operations" most - it is the biggest bummer in FE rendering in general. We'll try to finally work on this in v12 and will handle this with dedicated issues.
Updated by Benni Mack over 2 years ago
Christian Kuhn wrote in #note-53:
Hey. I'm gonna close this for now: It is known that menu rendering suffers from RootlineUtility "tree operations" most - it is the biggest bummer in FE rendering in general. We'll try to finally work on this in v12 and will handle this with dedicated issues.
Just as a side note, it ain't "just" the RootlineUtility. It's also the fact that HMENU is caching the "fully cached HTML output" separately per page (with active states etc) instead of caching it before.
Updated by Simon Schaufelberger over 2 years ago
I've been thinking about this also for some time and I think it would make sense to make the caching entries more reusable, this makes probably a huge impact for menu states like IFSUB, ACTIFSUB and CURIFSUB. When these states are used, there are quickly hundreds of SQL queries executed for big page trees. I have not yet explored the code of https://github.com/b13/menus but what does this extension make different?
Updated by Sybille Peters almost 2 years ago
@Christian Kuhn I would appreciate if we could keep this issue open (unless of course the performance has already been improved in v12. I must admit I have not followed the development that much lately).
For 2 reasons:
- there are a bunch of watchers in this issue, if this issue has a review on Gerrit one would be notified and could follow the development (and possibly also help with tests etc.)
- If you open the issue (e.g. based on an Internet search), you can immediately see the status - has it been improved already or not and for which TYPO3 versions?
- Stuff that belongs together is referenced and linked together.
- Sometimes people add some tips / workarounds / suggestions for improvements to issues. So until something is actually resolved, an issue can be a helpful resource.
(ok, more than 2, sorry).
I am currently researching solutions to improve performance for my menu (in TYPO3 v11) and there are some resources around and I have already signicantly improved load time with a simple fix. A search also returns this issue. Would be cool if one could find some more tips / workarounds here - also for versions before v11. (not sure if that is intended to misuse issues as knowledge database for fixes, but it is often used this way)
Updated by Benni Mack almost 2 years ago
- Related to Bug #98964: Menu object caching creates too many records resulting in huge cache_hash table added