Epic #88474
closedPage tree performance in 9.5
100%
Description
The new page tree delivered in v.9.5 has performance issues with bigger instalations e.g. 100k pages
Because of processing the whole page tree in one go.
We should make it possible to fetch one page tree level at once and process only these pages on PHP side.
For the discussion please join the slack channel
#page-tree-performance
https://typo3.slack.com/messages/CK20S7EKS
Updated by Wolfangel Cyril about 5 years ago
Two idéas :
If i remember well, the 4.5 -> 6.2 team already addressed a similar issue and worked on the pagetree performance. It's probably a regression.
Other thing, maybe we could remove the loading animation would immediatly solve the bug as the 7.X version is doing the action in the background and nobody complained ?
Updated by Johannes Wagner about 5 years ago
I have a large and deeply nested page tree. I want to install it in Typo9.5.
You could build the page tree asynchronously as in the previous Typo3 version. There, my page tree is quickly set up using the getNextTreeLevel method.
related question: https://stackoverflow.com/questions/56628254/how-can-the-typo3-backend-page-tree-be-constructed-asynchronously-in-typo3-v9-x
Updated by Sybille Peters almost 5 years ago
- Sprint Focus set to On Location Sprint
Issues with performance for pagetree can be reproduced with less than 100 000 pages. Of course, this depends on resources used. But a page tree of about 25 000 pages can make backend pretty unmanageable requiring several seconds wait time after inserting a new page, changing the page title or just logging in and trying to access the page tree.
I tested this with an initial 9.5.9 installation with Introduction Package where about 23 000 pages were added (recursively).
A lot of time is spent in here:
TreeController::fetchDataAction():
...
foreach ($entryPoints as $page) {
$items = array_merge($items, $this->pagesToFlatArray($page, (int)$page['uid']));
}
or specifically in the pagesToFlatArray function.
The items array is filled for all existing pages.
Loading the pages asynchronously would help a lot. If most parts of the page tree are collapsed, the page information is not really required (yet).
It also seems that costly operations are performed for minor changes, e.g. changing the page title of one page and saving.
Updated by Susanne Moog over 4 years ago
- Sprint Focus deleted (
On Location Sprint)
Updated by Markus Klein over 4 years ago
- Related to Task #90104: Introduce prepared statement for BackendUtility::getPageForRootline added
Updated by Markus Klein over 4 years ago
- Related to Task #90105: Allow full row to be passed into isInWebMount() added
Updated by Markus Klein over 4 years ago
Updated by Marko Nix over 4 years ago
I can add some investigation on that issue I think.
Setup¶
- TYPO3 v9.5.13 and for testing switched to 9.5.14-dev, all results remain the same.
- About 23k entries in pages table.
- Normally there is "be_acl" loaded and used, for investigation deactivated the extension.
- There are granular permissions set normally, to find an error all pages got an owner (we call him "owner") and is assigned to group "editors" with the permission bits: user 31, group 1, all 1.
- All editors have the root page as mount point and additional mount points for pages where they have write access. Everything else is read-only... "Page show" for them.
Problem¶
Page tree loading runs into maximum execution tim (set to 240 seconds).Using Backend user module to switch over to
- "owner": no page tree
- any "editors": no page tree
As administrator there is no problem, page tree loads immediately.
Ideas for solving - tried patches¶
https://review.typo3.org/c/Packages/TYPO3.CMS/+/62904 and https://review.typo3.org/c/Packages/TYPO3.CMS/+/62868 (same content?)
https://review.typo3.org/c/Packages/TYPO3.CMS/+/62867
https://review.typo3.org/c/Packages/TYPO3.CMS/+/62086
Some looks in the code behind¶
typo3/sysext/backend/Classes/Controller/Page/TreeController.php (approx line 671)
Changing
$entryPoint = $repository->getTree($entryPoint, function ($page) use ($backendUser) {
// check each page if the user has permission to access it
return $backendUser->doesUserHaveAccess($page, Permission::PAGE_SHOW);
});
to
$entryPoint = $repository->getTree($entryPoint, function ($page) use ($backendUser) {
// check each page if the user has permission to access it
return true;
});
"fixes" the problem, page tree loads immediately.
Digged a bit deeper and found the point where it may ends, but as far as i Know that is already known.
typo3/sysext/core/Classes/Authetication/BackendUserAuthentication.php (approx line 671)
if ($id > 0) {
$wM = $this->returnWebmounts();
$rL = BackendUtility::BEgetRootLine($id, ' AND ' . $readPerms);
foreach ($rL as $v) {
if ($v['uid'] && in_array($v['uid'], $wM)) {
return $v['uid'];
}
}
}
As far as i can see for all 23k pages the complete rootline is loaded. I figured out that one number can change everything. But at that point I am out, I have no idea to "fix" it.
typo3/sysext/backend/Classes/Utility/BackendUtitlity.php (approx line 383)
$output = $beGetRootLineCache[$ident];
} else {
$loopCheck = *100*;
$theRowArray = [];
while ($uid != 0 && $loopCheck) {
Changing following let me load an incomplete page tree in a normal time.
$output = $beGetRootLineCache[$ident];
} else {
$loopCheck = *3*;
$theRowArray = [];
while ($uid != 0 && $loopCheck) {
May this helps.
Updated by Uwe Trotzek over 4 years ago
I'm currently upgrading an 8.7 instance to version 10.2 and I'm also experiencing the described performance issues with the page tree.
I have a page tree with ~11k pages with a max nesting level of 7 levels.
Even with the applied patch of https://review.typo3.org/c/Packages/TYPO3.CMS/+/62086 the page tries to validate the permissions of every page.
After debugging a bit I could find out that my setup obviously completely bypass the "load pages on demand solution".
/**
* Returns JSON representing page tree
*
* @param ServerRequestInterface $request
* @return ResponseInterface
*/
public function fetchDataAction(ServerRequestInterface $request): ResponseInterface
{
$this->initializeConfiguration();
$items = [];
if (!empty($request->getQueryParams()['pid'])) {
// Fetching a part of a page tree
$entryPoints = $this->getAllEntryPointPageTrees((int)$request->getQueryParams()['pid']);
$mountPid = (int)($request->getQueryParams()['mount'] ?? 0);
$parentDepth = (int)($request->getQueryParams()['pidDepth'] ?? 0);
$this->levelsToFetch = $parentDepth + 2;
foreach ($entryPoints as $page) {
$items = array_merge($items, $this->pagesToFlatArray($page, $mountPid, $parentDepth));
}
} else {
$entryPoints = $this->getAllEntryPointPageTrees();
foreach ($entryPoints as $page) {
$items = array_merge($items, $this->pagesToFlatArray($page, (int)$page['uid']));
}
}
return new JsonResponse($items);
}
In my setup the condition "!empty($request->getQueryParams()['pid'])" is always false, which leads to a loading of the entire pagetree. I don't know when the parameter "$request->getQueryParams()['pid']" is set.
I can provide some details to the setup which we are using:
- We have only ONE mount point for all users. They all start at a root page.
- To restrict access to certain subpages, we configure the visibility for certain pages via the access module only for a defined group
- User who should "see" a page, are member of one or more groups.
I don't know if the "one mount point for all" setup is exotic, but this worked for us for many years in a bunch of projects.
Updated by Markus Klein over 4 years ago
Please comment your findings about a certain patch directly on Gerrit to the code. Thanks.
"one mount point for all" is definitely not exotic.
Updated by Tymoteusz Motylewski over 4 years ago
@Uwe
Did you make sure your browser is not caching the JS files?
The JS prt was modified to pass the pid param.
Please ensure you see the changed js file in your browser.
Updated by Uwe Trotzek over 4 years ago
Sorry, I was not at the office yesterday.
I just "moved" my comment to gerrit. Thanks for your responses so far.
Updated by Christian Kuhn about 4 years ago
- Related to Feature #90931: Add option to disable version highlighting in the page tree added
Updated by Christian Kuhn about 4 years ago
- Related to deleted (Feature #90931: Add option to disable version highlighting in the page tree)
Updated by S P about 4 years ago
- Related to Bug #91883: Filtering in the page tree shows translated pages alongside the L=0 ones added
Updated by S P about 4 years ago
- Related to Bug #91884: Page tree filter has no delete "X", needs hitting enter and has no filter info box anymore, a big backwards step regarding usability added
Updated by Anonymous about 4 years ago
- Related to Bug #91885: Page-Tree filter does not reset added
Updated by Anonymous about 4 years ago
- Related to Bug #91887: Page-Tree filter does not indicate its state added
Updated by Oliver Hader about 4 years ago
- Related to Bug #91902: Page tree not expanding deeper from initial view for editors added
Updated by Danilo Caccialanza about 4 years ago
- Related to Bug #91728: Very slow tree loading if you have multiple workspaces TYPO3 v10 added
Updated by Riccardo De Contardi almost 4 years ago
- Related to Bug #90056: Selection not open after after leaving filtered mode added
Updated by Riccardo De Contardi almost 4 years ago
- Related to Bug #92027: Reset page tree filter after deletion of page record added
Updated by Riccardo De Contardi over 3 years ago
- Related to Epic #93547: Collection of problems with large sites added