TYPO3 Core

Thanks for bringing this up. I also have the impression that menu rendering takes quite long. I also started looking around a bit but I did not find a bottleneck. The process simply seems to be complex (access checking, translation checking etc.).

But it would be really cool if someone has an idea. Especially for rendering sitemaps the slow menu rendering is a real pain.

In case someone has more experience analyzing profiler data, I conducted 3 test cases using the xdebug profiler:

0) without profiler, cleared FE caches
0) without profiler, reloaded news list to fill general caches
1) with profiler, loaded single news object
2) with profiler, loaded other single news object
3) with profiler, loaded calendar page

1) https://dl.dropboxusercontent.com/u/70926418/xdebug.local.out.1
2) https://dl.dropboxusercontent.com/u/70926418/xdebug.local.out.2
3) https://dl.dropboxusercontent.com/u/70926418/xdebug.local.out.3

1) https://dl.dropboxusercontent.com/u/70926418/xdebug.staging.out.1
2) https://dl.dropboxusercontent.com/u/70926418/xdebug.staging.out.2
3) https://dl.dropboxusercontent.com/u/70926418/xdebug.staging.out.3

So, we found out that indeed the performance problem has to do with too much database access. Finally, after searching for hours, I almost accidentally looked into the configuration module and found out, that even though apc was enabled in the install tool, it was not used anywhere.

Adding the following lines to typo3conf/AdditionalConfiguration.php disabled most cache to the database and now page rendering times dropped to a 10th! This is now acceptable.

$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_pages']['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\ApcBackend';
$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_pages']['options'] = array();

$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_hash']['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\ApcBackend';
$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_hash']['options'] = array();

$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_pagesection']['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\ApcBackend';
$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_pagesection']['options'] = array();

$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_rootline']['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\ApcBackend';
$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['cache_rootline']['options'] = array();

$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['extbase_reflection']['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\ApcBackend';
$GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations']['extbase_reflection']['options'] = array();

The question remains though, why does enabling the APC backend in the install tool not affect TYPO3 default caching behavior?

Hi Florian,

thank you very much for your research.

Where do you set the APC cache in the install tool? I know one setting under "Configuration Presets" but this only changes the cache type for the extbase_object cache.

I'm not sure which of your caches improves the menu rendering performance. Or does it only improve the performance for cached pages?

As far as I can see most of the menu rendering is done in the \TYPO3\CMS\Frontend\Page\PageRepository (e.g. ->getMenu()) which only uses direct database queries or its local caches. So I can not see how changing the configuration for all the other caches can improve the rendering times for menus.

Cheers,
Alex

Another thought: I'm wondering if the performance could be improved using MySQL indexes for the fields queried by the PageRepository, e.g. deleted, hidden, starttime, endtime, fe_group and optionally the versioning fields.

Where do you set the APC cache in the install tool? I know one setting under "Configuration Presets" but this only changes the cache type for the extbase_object cache.

I set it in the install tool only, it only changed extbase_object cache. Now I have this file in AdditionalConfiguration as I see no benefit at all for storing anything in database:

if (!function_exists('apc_for_cache_backend')) {
    function apc_for_cache_backend($cacheName)
    {
        $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations'][$cacheName]['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\ApcBackend';
        $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations'][$cacheName]['options'] = array();
    }
}

apc_for_cache_backend('cache_pages');
apc_for_cache_backend('cache_hash');
apc_for_cache_backend('cache_pagesection');
apc_for_cache_backend('cache_rootline');
apc_for_cache_backend('extbase_reflection');
apc_for_cache_backend('cache_news_category');

I'm not sure which of your caches improves the menu rendering performance. Or does it only improve the performance for cached pages?

I checked on cache performance once more by resetting the "cache_hash" table to database backend.
The rendering times increased to 4 seconds once more, so I believe the bottleneck is with this cache table.

I can fully confirm this report on TYPO3 CMS 6.2.4.

• The effect is NOT visible if your DB is running from an SSD, so you need a classical HD based MySQL server to reproduce
• expAll = 1 must be used to force the rendering of the complete menu tree
• you don't need a very big tree but you need some nesting, and ... very important ...
• the MenuItem depth must be reasonably big (e.g. 2 < .1, 3 < .1, 4 < .1 - see example below).

Here is a quick sketch of a minimal HMENU to get the effect.

page.10 = HMENU
page.10 {
    1 = TMENU
    1 {
        expAll = 1
        wrap = <ul>|</ul>
        NO = 1
        NO {
            wrapItemAndSub = <li>|</li>
        }
    }
    2 < .1
    3 < .1
    4 < .1
}

With a small page tree of some 30 pages (total page count over all levels) nested three levels deep, I get rendering times of 10 to 15 seconds per page (with pages that contain nothing but the above menu)! After a page has been rendered once and has been stored in cache, the next call serves the page reasonably fast. But with only 30 pages you can easily calculate that the total rendering time for all pages on first hit sums up to a total of 5 minutes. This is beyond any reasonable figure, especially considering that the pages are practically empty (only a menu contained).
I have tested all of this on two server systems with reasonable performance, one of them being a DomainFactory (Dedicated Server, nearly empty) which has always been a great and speedy hoster for us.
My testing CMS is a fresh and empty TYPO3 installation. On DomainFactory I have even used their TYPO3 default installation to test.

My conclusion is that there is some major problem in the back-ported caching framework. As a side note, I also have rendering times of about 15 seconds for a simple Extbase phone list extension in another project. That phone list has only one person table with 600 records and two 1:n tables for person attributes (company and department, about 20 records each). My profiling trials lead to the same conclusion: The caching framework eats up unbelievable resources.

Considering the ramifications of both effects (menu performance problems and extbase extension performance problems) and both issues obviously originating from the caching framework, I have to conclude, that 6.2 is simply not yet production ready. Unfortunately the problem is buried so deep in the caching system that I feel unable to debug it myself and present a solution (as I have done so many times before for TYPO3 over the last 10 years).

Hi everyone.

I have here the same problem.

I have the 6.2.4 version with the demo pages from the bootstrap_package. I watch the rendering time via admintool, and the results are this:

I take this page (not cashed) as example:

/about/history.html

The HMENU is called 6 times, (that seems OK) but this times:

10 HMENU 1364 1493 +12 =1505
10 HMENU 2870 653 +17 =670
10 HMENU 3541 1174
10 HMENU 4779 891
20 HMENU 5704 3381
10 HMENU 9089 148

Only the HMENU costed here 7.611 sec.

The times are every time a different, but nearly every time the HMENU costs between 50% and 75% of the whole rendering time.

I was now over one week looking for the bottleneck, and fond two points where the speed is going away in typo3.

The second was in the view, if you have a loops or similar elements, which you use to build for example tables, edit forms or something like that, for example:

<f:for each="{stores}" as="store">

... here some needful table rows ...

</f:for>

If you have both things HMENU and a plugin, which use f:for you get loading times up to:

-->> 15 sec.

maybe on servers with a better setup, SSHDs and with workarounds You get some better results. but every page with a simple navigation and a table with 20 rows which need longer then 0.3 - 0.5 sec is in my opinion google-dead; and that is for me a K.O. criterion ,... :-(

please help.

best regards

Nik

I can confirm this problem.
I have a website with about 120 pages on 4 levels.
The rendering of an uncached page takes 5000ms, the HMENU takes 3500ms of them, so about 70%.

TYPO3 6.2.6

I have the same problem. On Typo3 6.2.9, a website with about 150 pages on 4 levels is very slow rendering the uncached pages.
When deactivating the menus, page rendering time improves about 3000%. When page is cached, performance is ok, but the initial loading takes endlessly.

For testing purposes i created a special page type, which returns the menu (4 levels) only - it takes about 5 sec. to render on a hetzner managed server q90, which is a quite powerful machine. Other Typo3 projects on the same machine with a similar number of pages, but less menu levels are delievered in a blink of an eye

I can confirm this problem.
On Typo3 6.2.9, on a site with about 150 pages and a three level Menu HMENU rendering takes about 12 seconds according to the profiler extension. The configuration is simuliar to the other examples on this page.
Once the page is cached the performance is ok. The server is a root server with a very low load running PHP 5.5.19 with memory_limit set to 256M.

Hi Alexander,

all you need is:
– a page tree with some 30 empty pages nested 4 to 5 levels deep (quick to build with the Functions module and some page dragging to nest the pages)
– a HMENU of this form:

page.10 = HMENU
page.10 {
    1 = TMENU
    1 {
        expAll = 1
        wrap = <ul>|</ul>
        NO = 1
        NO {
            wrapItemAndSub = <li>|</li>
        }
    }
    2 < .1
    3 < .1
    4 < .1
    5 < .1
}

Thanks for looking into this!
Cheers, Jörg.

SSD or not is not the question. I will take a look into all SQL Queries that will be executed.

I upgraded one installation of Typo3 6.2.7 to Typo3 7 because of this problem.
It looks like the bug was fixed. Probably we only need a back port of the chaching changes?

I have the same problem with Typo3 6.2.9. Rendering of my main menu based on HMENU takes more than 10 seconds.
APC saved my day (see above) and rendering performance is ok now.

If I can help anyone fixing this problem feel free to contact me.

Merry Christmas
Florian

Could be related to: https://forge.typo3.org/issues/63550

Ran into some performance issues with a TMENU using expAll = 1 where empty menu objects were stored in cache (cache_hash) as NULL. The cache was checked using 'is_array' so the menu object was regenerated.
On some pages generating the page took 5 sec, now dropped to < 1s.

Also check if $GLOBALS['TYPO3_CONF_VARS']['FE']['activateContentAdapter'] is still activated which results in providing a fallback for the 'media' field from the 'pages' table and generating numerous additional queries for menu items.

@Arjen Hoekema:
A) #63550 sounds very promising. This could very well be the source of the issue here!
B) ['FE']['activateContentAdapter'] is FALSE in all of our environments. This was not the reason for the problem in our tests.

@Florian Kuss:
This has nothing to do with APC or any other OPCache system. We use OPCaches in all our live systems and still have the problem.

Hello,

is it sure that caching helps out here? Sure - when the cache works correcty (#63550) it will be much faster.
But I do not understand why generating the MENU itself takes such a long time, regardless of the cache?

Sebastian

Depending on the number of pages a huge number of SQL-queries are fired. Last time I looked there was a query (or two?) for every page being displayed in the menu. That might be related to checks for visibility or so.

I've applied the #63550 patch but clearly it doesn't fix the problem at all. This bug is very annoying in our bigger Typo3 instances. Looked serveral hours, debugging, writing emails to customers... I don't understand why this bug doesn't affect so much people. In Typo3 7 the bug was fixed, but where?

I think not everyone is affected by this bug because it heavily depends on your database / IO performance.

When your storage performance is not optimal the impact of this problem seems to be much bigger.

When you have fast harddisks (SSDs) you will not notice this problem so much.

At least these are my experiences.

We run MariaDB on huge DB servers, they arent slow... But the slowquery log shows very slow querys (1-5s) for INSERT statements in the cf_cache_hash.

confirmed

Actually we´re updating round about 30 Typo3-Services from 4.5 to 6.2 LTS but can´t explain these performance problems to our customers.

A pulldownmenu with 250 entries in 3 menulevel will slow down the service to 20 seconds response time without any content on the site.
This is a Root-Server with SSDs.

A sulution is needed fast as possible, we can´t work with typo3 6 since it´s fixed.

We are using a redis cache as a workaround. This configuration improved the speed for us quite a bit:

if (
    isset($GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['myext']['enableRedisCaches'])
    && $GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['myext']['enableRedisCaches']
    && extension_loaded('redis')
) {

    if (!function_exists('configureRedisAsCacheBackend')) {
        function configureRedisAsCacheBackend($cacheName)
        {
            $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations'][$cacheName]['backend'] = 'TYPO3\\CMS\\Core\\Cache\\Backend\\RedisBackend';
            $GLOBALS['TYPO3_CONF_VARS']['SYS']['caching']['cacheConfigurations'][$cacheName]['options'] = array();
        }
    }

    configureRedisAsCacheBackend('cache_pages');
    configureRedisAsCacheBackend('cache_hash');
    configureRedisAsCacheBackend('cache_pagesection');
    configureRedisAsCacheBackend('cache_rootline');
    configureRedisAsCacheBackend('extbase_reflection');
    configureRedisAsCacheBackend('cache_news_category');
}

We skippt the Menu generation with Typoscript and used a small Fluid Template to generate the Menu. This helped speeding up the rendering process a lot.
It is a workaround, but no direct solution for this problem.

Clemens Tilgner wrote:

We skippt the Menu generation with Typoscript and used a small Fluid Template to generate the Menu. This helped speeding up the rendering process a lot.
It is a workaround, but no direct solution for this problem.

How do you do that? VHS viewhelper?

If the menu is the same on all pages, you can cache it so it will only be rendererd on the first accessed page (taking the performance-penalty) and later be fetched from cache for all other pages that are generated.

One possibility for caching is for example stdWrap.cache: http://docs.typo3.org/typo3cms/TyposcriptReference/Functions/Cache/Index.html

Jörg Wagner wrote:

How do you do that? VHS viewhelper?

Correct. This is the template we used:
page.10.marks.NAVI = FLUIDTEMPLATE
page.10.marks.NAVI {
file = fileadmin/templates/fluid-templates/mainmenu.html
}

mainmenu.html: {namespace v=FluidTYPO3\Vhs\ViewHelpers}

<v:page.menu pageUid="1" expandAll="1" useShortcutUid="0">
<ul id="nav">
<f:for each="{menu}" as="menuPage">
...
</f:for>
</ul>
</v:page.menu>

Any update for that issue? It really is a pain right now for some of our installations with 2 or more levels in the HMENU. We have 6.2.11 running.

We can confirm this issue with 6.2.11 which includes #63550. on a large site with hundreds of pages the generation of an empty page with just the menu takes over a minute.

it is considerably better with 6.2.12 but it is still slow, about ~12 seconds. 4.5 LTS was able to do that in ~3 seconds. (tested with the same page, one upgraded one old, on the same server on two different ports)

Hi, we have copied our Typo3 4.5 to a test environment to Typo3 6.2.13. We use a nested HMenu (3 levels) and about 120 Pages. Before 6.2.12 it took about 20-30 seconds minimum to load a page without cache, with 6.2.13 it is now much better and takes about 7-12 seconds to load completely. We tried to optimize the TS as much as possible, removing conditions (except language conditions) and using less extensions. I even removed the ACT-state from the HMENU (so the menu basically always stays the same and is supposed to load very quick).

Everything else loads quick if I take a look at the adminpanel, only the language HMENU and the navigation HMENU take 1-2 seconds and 5-11 seconds respectively. The more or less same installation on 4.5 loads without cache in about 3 seconds.

I initially tried to convince my client to update to 6.2 because it is the new LTS with all its benefits, but the initial loading time is not acceptible for him (at least it is still quick with static cache after the first loading). Is the only solution putting the database to a SSD?

Is the only solution putting the database to a SSD?

As mentioned above there is also the option to use VHS viewhelper.

But our solution for now is that we have moved all database storage files to SSDs on our public servers. Luckily this is possible with our hoster at reasonable cost. This completely irons out the performance issues of TYPO3 6.2 for all our hosted customers and cured a lot of headaches.

Anyway, it is quite embarrassing that such measures are still necessary 15 months after launch of 6.2.

Jörg Wagner wrote:

Is the only solution putting the database to a SSD?

As mentioned above there is also the option to use VHS viewhelper.

Hi Jörg, thanks for the tip, It took me awhile to be able to create a 3 level columnar dropdown menu. Here is my code, if it might help someone. I had to adapt the css since I couldn't find out how to adapt Typoscript's wrapItemAndSub correctly with fluid. The Website loads faster now, 6-8 seconds and without spikes (with HMENU it sometimes took anything between 12 and 50 seconds).

{namespace v=FluidTYPO3\vhs\ViewHelpers}
<v:page.menu useShortcutData="TRUE" pageUid="264" expandAll="1" levels="3">
<ul class="menu">
  <f:for each="{menu}" as="item">
  <li>
    <f:link.page pageUid="{item.uid}" title="{item.linktext}">{item.linktext}</f:link.page>
    <f:if condition="{item.hasSubPages}">

      <v:page.menu pageUid="{item.uid}">
      <ul class="second">
        <f:for each="{menu}" as="subItem">
        <li>
          <f:link.page pageUid="{subItem.uid}" title="{subItem.linktext}">{subItem.linktext}</f:link.page>
            <f:if condition="{item.hasSubPages}">

              <v:page.menu pageUid="{subItem.uid}">
              <ul class="third">
                <f:for each="{menu}" as="subsubItem">
                <li>
                  <f:link.page pageUid="{subsubItem.uid}" title="{subsubItem.linktext}">{subsubItem.linktext}</f:link.page>
                </li>
                </f:for>
              </ul>
              </v:page.menu>

            </f:if>
        </li>         
        </f:for>
      </ul>
      </v:page.menu>

    </f:if>
  </li>
  </f:for>
</ul>
</v:page.menu>

Instead of nesting all the levels, a main template with section would be helpful in this case which calls itself. Something like this (from Neos):

https://git.typo3.org/Packages/TYPO3.Neos.NodeTypes.git/blob/HEAD:/Resources/Private/Templates/NodeTypes/Menu.html

You don't need to repeat the menu for every level, just use <v:page.menu.sub>

Beside the solutions with fluid or use of APC, there's another option to speedup big HMENUs with expAll: use the possibility of caching.
Normally, big parts of these menus does not change on each page, but the whole HMENU is rendered per page. Why not using identical parts of the menu for different pages? All inactive branches of the menu could be cached. This lowers the count of database queries.

For details of such a solution, there's a blogpost: http://stichwort.org/typo3/typoscript/mega-menue-mit-cache-optimieren/ (German only)

We've written an alternative, based on the addition to the blog post of the last comment, made on 2018-05-17:
Addition in the blogpost: https://gist.github.com/pgampe/cb29bc0fc1111d1370cc
Our generalized approach, based on that: https://gist.github.com/julrich/d91c1ac1968418e6ae8290f2a2e90afb

Maybe this is helpful for somebody else!
Handles cases like the special pages (index page, search page, pages not inside the root node of the menu, where the replacement RegExp otherwise would be an empty string, replacing everything), logged in users and languages.

Btw. what make you say this: "should be solved now for versions 7 and 8", Riccardo? At least for us, it wasn't (8.7).

@Jonas Ulrich basically there has not been much activity on this issue; I guessed it was solved because of the lack of people complaining about it... It it absolutely possible that I am wrong. I also tried an empiric test 9 months ago and had no feedback...

Please, if you have more information about how to reproduce the problem write it; I'll reopen it if you want.

Regards,

Riccardo

@Jonas

1) Please edit your message and use the thumbnail feature to avoid this huge image inline.
2) Please add the corresponding typoscript which generated this graph.

Thanks.

I've changed the image to use the thumbnail, sorry about that.

As our menu is quite complex, I can't really rip it out (multiple .ts-files) easily.
I will see (can't promise anything) if I can add a reduced example. But to be honest, to me it seems that the problem is just unchanged.

I'd hazard a guess, but the minimal example, posted by Jörg Wagner 3 years ago, should also still exhibit those performance penalties for large(r) menus:
https://forge.typo3.org/projects/typo3cms-core/activity?from=2014-12-17

Christian Kuhn wrote in #note-53:

Hey. I'm gonna close this for now: It is known that menu rendering suffers from RootlineUtility "tree operations" most - it is the biggest bummer in FE rendering in general. We'll try to finally work on this in v12 and will handle this with dedicated issues.

Just as a side note, it ain't "just" the RootlineUtility. It's also the fact that HMENU is caching the "fully cached HTML output" separately per page (with active states etc) instead of caching it before.

I've been thinking about this also for some time and I think it would make sense to make the caching entries more reusable, this makes probably a huge impact for menu states like IFSUB, ACTIFSUB and CURIFSUB. When these states are used, there are quickly hundreds of SQL queries executed for big page trees. I have not yet explored the code of https://github.com/b13/menus but what does this extension make different?

@Christian Kuhn I would appreciate if we could keep this issue open (unless of course the performance has already been improved in v12. I must admit I have not followed the development that much lately).

For 2 reasons:

• there are a bunch of watchers in this issue, if this issue has a review on Gerrit one would be notified and could follow the development (and possibly also help with tests etc.)
• If you open the issue (e.g. based on an Internet search), you can immediately see the status - has it been improved already or not and for which TYPO3 versions?
• Stuff that belongs together is referenced and linked together.
• Sometimes people add some tips / workarounds / suggestions for improvements to issues. So until something is actually resolved, an issue can be a helpful resource.

(ok, more than 2, sorry).

I am currently researching solutions to improve performance for my menu (in TYPO3 v11) and there are some resources around and I have already signicantly improved load time with a simple fix. A search also returns this issue. Would be cool if one could find some more tips / workarounds here - also for versions before v11. (not sure if that is intended to misuse issues as knowledge database for fixes, but it is often used this way)